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ABSTRACT 


This  document  describes  the  results  of  one  part  of  a  full-scale  system  test  of 
concepts  for  improvement  of  quality  and  of  display  of  Navy  system-related  Technical 
Information  developed  by  the  Navy  Technical  Information  Presentation  System  (NTIPS). 
The  tests  involved  comparisons  of  three  types  of  experimental  Technical  Information  (TI) 
with  the  conventional  paper  work  package.  The  three  types  of  experimented  TI  were 
(1)  NTIPS  automated  troubleshooting  TI  (called  Fault  Isolation  by  Nodal  Dependency, 
FIND),  electronicedly  displayed;  (2)  NTIPS  electronically  displayed  corrective- 
maintenance  TI;  and  (3)  NTIPS  corrective-maintenance  TI  delivered  on  paper. 

Tests  were  carried  out  at  Miramar  Naval  Air  Station  using  an  operational  F-14A 
aircraft  (with  introduced  "faults").  Test  subjects  were  squadron  Aviation  Electrician's 
Mates  (AEs)  both  experienced  and  inexperienced. 

All  test  objectives  were  achieved.  The  tests  demonstrated  that  electronic  display  of 
maintenance  TI  is  highly  acceptable  to  fleet  personnel  (9096  of  the  test  subjects  favored 
electronic  display),  that  NTIPS-proposed  modifications  to  TI  are  as  effective  or  more 
effective  than  conventional  TI  in  supporting  troubleshooting  and  corrective  maintenance, 
and  that  automated  troubleshooting  produced  a  highly  significant  improvement  in  fault- 
isolation  success. 

The  tests  provided  valuable  experience  and  indicated  areas  in  which  NTIPS  TI  and 
electronic  display  can  be  improved.  Proposals  resulting  from  the  test  are  currently  being 
implemented. 
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1.0  TEST  SUMMARY 


1.1  BACKGROUND  AND  PURPOSE 

This  report  describes  an  operational  field  test  of  the  Navy  Technical  Information 
Presentation  System  (NTIPS),  a  system  that  has  been  designed  to  improve  the  quality  of 
Technical  Information  (TI)  for  logistic  support  of  Navy  weapons  systems  and  to  reduce  the 
difficulty  and  expense  of  acquiring  and  managing  it.  NTIPS  maximizes  reliance  on 
automated  systems,  starting  with  documentation  specifications  and  authoring  procedures 
and  culminating  in  the  electronic  delivery  of  TI  to  the  technician  at  the  maintenance  site. 
NTIPS  is  currently  in  Phase  III:  Test  and  Evaluation.  During  the  spring  and  summer  of 
1986,  a  test  plan*  was  developed  for  comparing  TI  generated  with  NTIPS  procedures  and 
according  to  NTIPS  specifications  to  conventional  paper  worlc-package  Technicsd  Manuals 
used  to  support  maintenance  on  the  F-14A  fighter  aircraft.  A  pretest  of  the  plan  was 
conducted  in  September  at  the  Miramar  Naval  Air  Station,  with  the  full  test  occurring  in 
October.  The  objectives  of  the  field  test  were  as  follows: 

0  Compare  the  performance  of  enlisted  maintenance  technicians  using 
the  TI  prepared  under  NTIPS  procedures  with  the  performance  of 
technicians  using  conventional  TI  (the  paper  F-14A  Technical 
Manual). 

0  Compare  technicians'  performance  when  guided  by  TI  printed  on 
paper  versus  TI  presented  via  an  electronic  medium. 

o  Establish  which  design  characteristics  of  NTIPS  TI  are  most  effective 
or  least  effective  in  an  operational  situation. 

o  Assess  user  acceptance  of  certain  features  (medium,  content,  format, 
and  style)  of  the  NTIPS  TI  presentation. 

All  these  objectives  were  achieved.  This  section  provides  an  overview  of  the  test  design, 
test  executions,  and  test  results.  In  carrying  out  this  field  test  of  NTIPS  TI,  an  off-the- 
shelf  electronic  delivery  device  was  used.  The  test  was  designed  not  to  test  fielded 
hardware,  but  rather  to  test  NTIPS  approaches  to  (1)  creating  TI  that  is  expected  to  be 
intrinsically  more  effective  than  conventional  TI,  and  (2)  displaying  this  TI  electronically 
and  reducing  reliance  on  paper  manuals.  In  addition,  the  test  was  designed  to  provide 


*NAVY  TECHNICAL  INFORMATION  PRESENTATION  PROGRAM.  Phase  III  Test  and 
Evaluation  of  the  Navy  Technical  Information  Presentation  System.  F-14A  Experimental 
Technical  Information  Test  Plan.  September  1986  (Essex  Corporation) 
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guidance  in  establishing  areas  of  needed  improvement  to  the  TI  and  the  delivery  device,  as 
well  as  to  demonstrate  the  current  effectiveness  of  the  NTIPS  approaches. 


1.2  TEST  SrrS  AND  TEST  PERSONNEL 

The  field  test  was  conducted  at  NAS  Miramar,  California  in  the  maintenance 
training  areas  of  this  Air  Station.  The  test  utilized  a  fully  operational  F-14A  aircraft 
(from  VF-124).  The  subjects  were  active-duty  Aviation  Electrician's  Mates  (AE's)  made 
available  by  Commander,  Fighter  Airborne  Early  Warning  Wing,  Pacific 
(COMFIT AEWWINGP AC)  from  a  number  of  the  squadrons  under  his  command.*  These 
technicians  were  segregated  into  experienced  and  inexperienced  groups  based  on  the 
extent  of  their  relevant  hands-on  maintenance  experience  with  the  F-14A  and  on  the 
judgment  of  the  maintenance  Chief  from  VF-124  FRAMP,  the  Fleet  Replacement 
Aviation  Maintenance  Personnel,  the  training  organization  at  Miramar.  When  performing 
multiperson  tasks,  test  subjects  were  assisted  by  FRAMP  instructors.  These  instructors 
also  served  as  test  coordinators.  Other  test  personnel  included  a  Test  Director  and  video 
crew  from  the  David  Taylor  Naval  Ship  Research  and  Development  Center  (DTNSRDC), 
two  data  collectors  from  Essex  Corporation,  and  a  computer  specialist  from  Hughes 
Aircraft. 

1.3  SCOPE  OP  THE  TEST 

To  establish  the  test  conditions,  faults  were  deliberately  introduced  into  the  aircraft 
by  the  test  coordinators.  This  fault  insertion  permitted  the  test  to  include  performance 
of  both  troubleshooting  (fault  verification  followed  by  fault  isolation)  and  corrective- 
maintenance  ta3ks  on  the  Rudder  Manual  Trim  System  (RMTS)  of  the  F-14A.  A 
VIDS/MAF  (Visual  Display  Systems/Maintenance  Action  Form)  stating  "Rudder  Trim  Inop" 
was  used  to  initiate  test  subjects'  performance.  On  the  basis  of  this  standard  approach  to 
initiating  a  maintenance  action,  the  test  subject  was  asked  to  verify  and  isolate  the  cause 
of  the  fault  introduced  into  the  aircraft.  For  fault  isolation  (troubleshooting),  half  of  the 
test  subjects  used  the  NTIPS  electronic  display  system  (Fault  Isolation  by  Nodal 
Dependency  -  FIND),  while  the  other  half  used  conventional  paper  work-package 
Technical  Manual  material  containing  primarily  schematics. 


*  Participating  organizations  were:  NAMTRADET,  AIMD,  VF-123,  VF-114,  VF-24,  VF-211, 
VF-213,  VF-301,  VF-1,  VF-154,  VF-2,  VF-21. 
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In  order  to  provide  an  adequate  set  of  corrective-maintenance  tasks  for  test 
purposes,  experimental  TI  was  prepared  for  five  tasks: 

o  Remove  the  Directional  Trim  Actuator  (DTA),  a  component  of  the 
Rudder  Manual  Trim  System 

o  Reinstall  the  DTA 

o  Zero  the  Rudder  Protractor 

o  Attach  the  Rudder  Protractor 

o  Check  and  Adjust  Rudder  Positioning 

Three  types  of  TI  were  used  for  corrective  maintenance:  NTIPS  TI  presented  electron¬ 
ically,  NTIPS  TI  presented  in  a  paper  form,  and  conventional  paper  work-package 
Technical  Manuals. 

The  types  of  troubleshooting  and  corrective-maintenance  TI  were  compared  by 
evaluating  the  performance  of  technicians  using  each  TI  type.  Performance  measures 
included  time  required  for  successful  completion  of  the  tsisk  (performance  times)  and 
number  of  errors  committed  by  technicians  during  task  performance.  These  measures 
were  supplemented  by  the  subjects'  own  evaluations,  obtained  by  questionnaires,  of  the 
usefulness  and  effectiveness  of  each  type  of  TI  for  performing  both  troubleshooting  and 
corrective  maintenance. 


1.4  TEST  EVENTS 

After  the  experimental  Technical  Information  was  prepared  and  reviewed,  the  field 
test  consisted  of  the  following  events: 

1.  Dry  Run  of  Test  Scenario  at  NAS,  Miramsm,  2  May  1986.  Test  personnel 
observed  a  walk-through  of  the  test  procedures  by  an  inexperienced  AZ-3 
technician  to  establish  time  and  physical  relationship  factors,  assess  the 
need  to  make  changes  in  the  test  plan  prior  to  the  pretest  event,  and 
work  out  relationships  between  test  observers,  test  subjects,  and  the  test 
support  personnel  from  Miramar. 

2.  Pretest  Event,  9-11  September  1986.  During  this  event  two  AE-2s  and 
two  AE-3s  (three  of  four  with  more  than  1  year  of  experience  on  the  F- 
14 A)  performed  the  maintenance  tasks  planned  for  the  actual  field  tests. 

As  a  result  of  this  event,  a  number  of  changes  were  incorporated  into  the 
proposed  test  procedures,  and  final  scheduling  and  procedures  were 
established. 

3.  Field  Test,  14-24  October  1986.  Test  and  evaluation  of  technician 
performance  using  conventional  and  experimental  (NTIPS)  TI.  Tasks 
included  Fault  Verification,  Troubleshooting  (Fault  Isolation),  and 
Corrective  Maintenance. 
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1.5  CONDUCT  OF  THE  TEST  EVENTS  BT  ENLKTED  TECHNICIANS 

Subjects  for  the  test  consisted  of  24  enlisted  technicians,  12  experienced  and  12 
inexperienced.  After  an  instruction  session,  each  technician  performed  a  troubleshooting 
task  followed  by  a  corrective-maintenance  task,  under  observation  by  test  personnel  and 
by  a  senior  instructor  from  the  FRAMP.  The  work  of  each  subject  was  ordered  so  that  he 
used  TI  of  two  different  types  (e.g.,  troubleshooting  by  FIND  followed  by  conventional 
paper  TI  for  corrective  maintenance,  as  shown  later  in  greater  detail  by  Fig.  1). 
Performance  times  (broken  down  by  Individual  actions)  and  performance  errors  for  each 
type  of  TI  tested  were  recorded.  After  the  test,  preference  questionnaires  were  filled  out 
by  each  technician,  and  an  oral  debrief  was  conducted  to  elicit  comments  and  suggested 
improvements  for  TI  constructions  and  presentation.  These  results  are  all  detailed  in  the 
present  report. 


1.6  SUMMARY  OF  TEST  RESULTS 

All  test  objectives  were  accomplished.  Carefully  monitored  technician  performance 
using  NTIPS  TI  covering  several  types  of  tasks,  with  both  electronic-delivery  and  paper 
presentation,  showed  the  following  results: 

1.  For  troubleshooting  tasks.  Inexperienced  technicians  achieved  a  slightly 
lower  overall  performance  time  (8%)  using  FIND  (NTIPS  troubleshooting 
TI,  electronically  presented)  than  when  using  conventional  paper-based 
work-package  TM's.  Experienced  technicians  took  significantly  longer 
(37%)  to  accomplish  troubleshooting  tasks  with  FIND  than  with  conven¬ 
tional  troubleshooting  TI. 

2.  For  troubleshooting  tasks,  ^  technicians  who  used  FIND  successfully 
isolated  the  fault  without  instructor  assistance.  Of  the  seven  expe¬ 
rienced  technicians  who  performed  the  troubleshooting  task  using  con¬ 
ventional  TI,  two  failed  to  isolate  the  fault.  Of  the  five  inexperienced 
technicians  who  performed  the  troubleshooting  task  using  conventional 
TI,  all  five  failed  to  isolate  the  fault. 

3.  For  corrective-maintenance  tasks.  Performance  times  for  all  three  TI 
types  used  (conventional  paper,  NTIPS  paper,  and  NTIPS  electronic 
presentation)  were  approximately  the  same  for  both  experienced  and 
Inexperienced  technicians.  As  a  group,  the  inexperienced  technicians 
required  slightly  more  time  than  experienced  technicians  when  using 
either  conventional  paper  or  NTIPS  electronically  presented  TI. 

4.  For  corrective-maintenance  tasks.  Inexperienced  technicians  each  com¬ 
mitted  an  average  of  8.0  errors  when  using  conventional  paper  TI.  This 
was  reduced  to  5.3  errors/technician  using  NTIPS  TI  electronically 
delivered  and  5.2  errors/technician  using  NTIPS  TI  on  paper. 

5.  For  corrective-maintenance  tasks.  Errors  committed  by  experienced 
technicians  were  approximately  the  same  for  each  type  of  TI  (a  range  of 
3. 5-4.0  errors/technician). 
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6.  Of  the  20  test  personnel  who  used  both  electronically  presented  TI  and 
paper  TI,  18  (90%)  preferred  electronic  presentation.  Specifically,  after 
the  tests,  90%  of  the  test  subjects  answered  the  following  written 
question  by  choosing  electronic  presentations:  If  you  had  a  choice  of 
using  an  electronic  or  paper-based  manual  to  perform  tasks,  which  would 
you  choose? 

1.7  TEST  CONCLUSIONS 

With  the  enthusiastic  cooperation  of  COMFITAEWWINGPAC,  Squadron  VF-124,  and 
enlisted  instructors  of  the  FRAMP,  the  NTIPS  field  test  was  carried  out  successfully.  All 
test  objectives  were  fully  achieved.  Test  results  are  considered  to  have  demonstrated  the 
following  conclusions: 

1.  Technical  information  presented  electronically  represents  a  distinct 
improvement  in  the  eyes  of  fleet  technicians  engaged  in  maintenance  of 
operational  aircraft.  (This  result  was  obtained  even  with  the  use  of  off- 
the-shelf,  non-portable  equipment  which  was  not  optimally  designed  for 
operational  use  and  with  graphics  which  were  clearly  capable  of  much 
improvement.) 

2.  NTIPS  presentations  of  troubleshooting  TI,  if  carried  out  according  to 
specifications,  can  result  in  significant  improvements  in  fault-isolation 
effectiveness. 

3.  The  NTIPS  presentation  methods  are  effective  in  improving  the  per¬ 
formance  of  inexperienced  personnel. 

4.  Valuable  recommendations  were  provided  by  test  personnel  and  FRAMP 
instructors  during  the  tests.  For  example,  it  was  made  clear  that  the 
quality  of  the  graphics  in  any  future  uses  of  electronic  presentation  must 
be  much  improved  over  that  of  the  graphics  used  in  the  test.  This 
improvement  would  involve  design  of  the  graphic,  graphic  size,  and 
graphic  resolution.  As  another  example,  flexibility  must  be  introduced 
into  the  automated  TI  presentations  to  permit  experienced  technicians  to 
move  more  rapidly  through  a  series  of  steps  without  the  time-consuming 
necessity  of  continually  viewing  material  they  already  know  from 
experience. 


Detailed  results  of  the  tests  and  specific  recommendations  of  the  test  personnel  are 
discussed  in  Section  4.0. 


1.8  ORGANIZATION  OF  TEST  REPORT 

Section  2.0  reviews  the  preparation  and  review  of  the  experimental  TI  (electronic 
presentation  and  paper)  used  for  the  test  and  describes  the  actual  tasks  selected  for 
testing.  Section  3.0  reviews  the  test  design,  which  was  independently  published  as  an 
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Essex  report  (cited  on  page  3).  The  initial  Test  Plan  was  modified  to  some  extent  as  a 
result  of  the  Dry  Run  and  of  the  Pretest  Trials  discussed  previously.  These  changes  are 
presented  in  Section  3.0.  Section  4.0  discusses  in  detail  the  test  results  and  summarises 
performance  times,  performance  errors,  uid  other  results  obtained  from  observing  the 
test  tasks  performed  by  technicians  using  the  five  kinds  of  TI  tested.  These  results 
include  the  subjects'  preferences  and  recommendations  for  improving  NTIPS  TI  and 
electronic  presentation.  Section  5.0  consists  of  a  summary  of  the  test  operations  and 
conclusions  resulting  from  it.  The  actual  forms  used  for  data  collection  are  incorporated 
as  Appendix  A.  The  preference  questionnaire  administered  to  the  24  test  subjects  is 
incorporated  as  Appendix  B.  Samples  of  the  TI  tested  are  included  in  Appendix  C. 
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2.0  PEBPARATION  AND  REVEBN  OF  EXPERIMENTAL  AND  CONVENTIONAL  TI 


2.1  INTRODUCTION 

This  section  discusses  the  generation  of  the  experimental  NTIPS  TI  (2.2),  the  review 
of  this  TI  for  test  suitability  (2.3),  and  the  review  of  conventional  TI  for  compatibility 
with  the  NTIPS  TI  (2.4). 

2.2  GENERATION  OF  EXPERIMENTAL  TI 

The  experimental  TI  was  prepared  by  Grumman  Aerospace  Corporation,  Bethpage, 
Long  Island,  New  York.  To  prepare  the  TI  for  test  purposes,  Grumman  was  provided  with 
(1)  a  MODCOMP  computer  which  hosted  an  automated  authoring  system  developed  under 
the  NTIPS  Program;  (2)  an  authoring  terminal;  (3)  a  screen  printer;  and  (4)  a  modem  which 
permitted  communication  with  a  similar  MODCOMP  at  Hughes  Aircraft  Company,  Long 
Beach,  California  for  obtaining  assistance  when  problems  arose.  Grumman  personnel  were 
trained  by  Hughes  personnel  in  operation  of  the  authoring  system  and  in  developing  system 
signal^ependeney  information  for  the  FIND  automated  troubleshooting  program. 

NTIPS  specifications  provided  to  Grumman  for  use  in  preparing  the  experimental  TI 
included  general  content,  format,  and  style  specifications  covering  the  following  TI 
characteristics: 

a.  Procedures 

b.  Descriptive  information 

c.  Illustrations 

d.  Style  (general) 

e.  Numbering,  indexing,  and  how-to-use  information 

f.  Diagrams. 

Also  provided  was  the  NTIPS  specification  entitled  Tault  Isolation  by  Nodal  Dependency 
(FIND):  Troubleshooting  Equipment,  Software,  and  Products”  to  permit  generation  of 
electronically  displayed  troubleshooting  information. 

Before  delivery  was  made  to  the  NTIPS  Office,  the  draft  TI  was  validated  with  the 
use  of  an  operational  F-14A  aircraft  at  Qrumman's  Calverton  facility  on  Long  Island. 
Validation  consisted  of  a  technical  accuracy  review  by  subject  matter  experts  and 
involved  comparing  the  TI  against  the  aircraft's  Rudder  Manual  Trim  System.  Reviews  to 
ensure  that  the  experimental  TI  (both  on  paper  and  in  electronic-display  form)  was  in 
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compliance  with  NTIPS  specifications  for  content,  format,  and  style  were  performed  by 
Grumman,  by  Hughes  Aircraft,  by  Essex  Corporation,  and  by  the  NTIPS  Office. 

Observations  made  by  Grumman  during  preparation  of  the  experimental  TI  were 
recorded  in  detail  in  a  Journal  format  (a  *liOg'*),  and  these,jiala-were  evaluated  to 
establish  the  possible  need  for  NTIPS  modifications,  '"^^tain  changes  to  the  NTIPS- 
designed  authoring  system,  to  the  electronic-display  system,  and  to  the  NTIPS  TI 
specification  have  already  been  carried  out  as  a  result  of  Grum  man's  experience.  The 
experience  of  Grumman  Aerospace  in  applyii^  for  the  first  time  the  NTIPS  TI  specifica¬ 
tions  and  in  using  for  the  first  time  the  automated  authoring  system  developed  under 
NTIPS  constituted  in  themselves  a  valuable  system  test,  which  demonstrated  the 
technological  feasibility  of  automated  preparation  of  TI  of  the  NTIPS  type. 

The  following  experimental  TI  was  generated  for  the  Rudder  Manual  Trim  System 
(RMTS)  tasks: 

o  A  fault-verification  procedure.  Verification  by  maintenance 
technicians  of  a  fault  reported  by  the  aircraft  crew  produces  the 
fault  symptoms  used  for  beginning  the  troubleshooting  (fault- 
isolation)  procedure. 

o  Troubleshooting  TI,  implemented  on  the  FIND  system,  used  to  isolate 
the  faulty  component(s)  producii^  an  RMTS  malfunction  (for  exam¬ 
ple,  the  Directional  Trim  Actuator,  Rudder  Surface  Position  In¬ 
dicator,  or  any  one  of  the  Pushrods).  The  only  delivery  medium  for 
FIND  TI  is  electronic, 

o  Corrective-maintenance  procedure  for  removal  and  reinstallation  of 
the  Rudder  Surface  Position  Indicator,  the  Directional  Trim  Actua¬ 
tor,  and  the  Pushrods.  In  addition,  check  and  adjustment  procedures 
were  provided  for  the  Directional  Trim  Actuator  and  any  of  the 
Pushrods. 

o  TI  procedures  (for  both  electronic  delivery  and  on  paper)  for  all 
supporting  tasks  involved  in  readying  the  aircraft  for  maintenance 
(such  as  Connect  and  Disconnect  External  Electric  Power)  and  in 
restoring  aircraft  to  a  ready  condition.  The  relevant  Illustrated  Parts 
Breakdown  (IPB)  was  also  provided. 

In  accordance  with  NTIPS  procedures,  Grumman  generated  the  above  experimental 
TI  In  a  single  electronic  data  base,  which  was  output  in  both  paper  and  electronic-delivery 
format  by  Hughes  (under  contract  to  the  NTIPS  Office)  during  the  mastering  process.  In 
the  test,  one  form  of  the  NTIPS  test  TI  was  delivered  electronically  by  an  AT&T  3B-2/300 
computer,  using  an  AT&T  6300  as  the  subject’s  interactive  terminal.  Test  subjects 
entered  commands  via  the  6300*8  touch  screen  or  keyboard.  The  display  system  weighs 
approximately  40  pounds  and  has  a  screen  size  of  6.5  x  7  inches  with  a  resolution  of  640  x 
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400  lines.  This  commercial  equipment  was  used  to  simulate  the  NTIPS  display  system  for 
flight-line  maintenance  which,  when  fielded,  will  be  a  portable  device  weighing  approx¬ 
imately  10  pounds,  with  dimensions  of  12"  x  9"  x  2",  with  a  screen  size  of  6.4"  x  9.6"  and  a 
resolution  of  840  lines  x  960  lines.  The  tests  were  thus  designed  to  test  the  TI  itself  and 
the  NTIPS  methods  of  presentation,  rather  than  any  hardware  device. 

Except  for  FIND  troubleshooting  TI,  the  TI  described  above  was  provided  on  paper 
medium  as  well  as  in  electronic-delivery  form. 

2.3  REVIEW  OF  EXPERIMENTAL  TI  BY  NTIPS  OFFICE 

The  final  form  of  the  experimental  TI  resulted  from  modifications  and  corrections 
based  on  a  detailed  review  performed  by  NTIPS  Office  personnel  and  other  Navy  and 
contractor  organizations  (Hughes  and  Essex  Corporation).  This  review  was  supplemented 
by  a  final  detailed  on-site  validation  at  Miramar  Naval  Air  Station  in  which  the 
procedures  and  graphics  to  be  used  in  the  tests  were  checked  in  final  form  by  FRAMP 
instructors  against  an  operational  F-14A.  Changes  suggested  as  a  result  of  this  review 
were  incorporated  into  the  final  NTIPS  TI.  These  reviews  established  that  the  experimen¬ 
tal  TI  was  in  accordance  with  NTIPS  Office  style,  content,  and  format  specifications; 
verified  that  the  experimental  TI  contained  all  information  needed  by  the  technicians  to 
perform  the  tasks  used  for  test  purposes;  and  verified  that  the  established  information 
was  accurate  and  was  presented  as  clearly  and  simply  as  possible. 


2.4  REVIEW  OF  CONVENTIONAL  TI  FOR  THE  CONTROL  TESTS 

Conventional  TI  normally  used  for  F-14A  maintenance  (that  is,  relevant  work 
packages  of  the  F-14A  Technical  Manuals  NAVAIR  01-F-14AAA-2-2-16.3,  NAVAIR  01-F- 
14AAA-2-3-4.3,  and  NAVAIR  01-F-14AAA-2-4-4.1)  was  used  for  all  troubleshooting  and 
corrective-maintenance  RMTS  tasks  involved  in  the  tests.  The  NTIPS  TI  and  conventional 
TI  were  compared  to  ensure  consistency  in  the  following  areas: 

o  Troubleshooting.  The  FIND  and  conventional  troubleshooting  mate¬ 
rials  cover  the  same  area  of  RMTS  hardware,  i.e.,  both  cover  the 
symptoms  and  test  points  for  the  faulty  component.  Specifically,  in 
both  sets  of  TI,  review  ensured  that  the  capability  existed  to  proceed 
logically  from  specific  trouble  symptoms  to  isolation  of  the  faulty 
component  producing  the  trouble  symptom. 

o  Corrective  maintenance.  Both  sets  of  TI  cover  the  same  corrective 
actions  (in  this  case  replacement  of  specified  components). 
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3.0  TB8TDE&9GN 


3.1  Test  Personnel 

The  NTIPS  F-14A  test  was  conducted  during  the  period  October  14  to  October  24  in 
close  coordination  with  the  Fleet  Replacement  Aviation  Maintenance  Personnel  (FRAMP) 
organization,  VF-124,  NAS,  Miramar.  Actual  performance  of  test  tasks  took  place  in  the 
FRAMP  maintenance  area  used  for  trainit^.  This  hangar  is  a  short  distance  from  a 
runway  which  permits  aircraft  to  be  brought  in  directly  from  the  Miramar  NAS  squadron 
areas.  A  listing  of  test  personnel  and  their  functions  is  shown  in  Table  1.  Test 
coordination,  subject  scheduling,  and  technical  consultation  were  provided  by  FRAMP 
management.  Instructor  personnel,  also  from  FRAMP,  conducted  subject  test  briefings, 
observed  and  evaluated  subject  performance,  and  served  as  helpers  to  subjects  when 
multiperson  task  performance  was  required.  Twenty-four  technicians  from  squadrons  of 
COMFIT AEWWINGP AC  participated  as  test  subjects.  These  subjects  performed  trouble¬ 
shooting  and  corrective-maintenance  tasks  as  specified  in  the  Test  Plan  (3.3),  responded 
to  a  preference  questionnaire  designed  to  evaluate  the  strengths  and  weaknesses  of  NTIPS 
TI  as  compared  to  conventional  paper  work  packages,  and  offered  comments  during 
videotaped  debriefings. 

The  test  was  supervised  by  a  Test  Director  from  DTRC.  The  functions  of  the  Test 
Director  were  to  coordinate  the  test  with  personnel  of  NAS,  Miramar;  to  provide 
technical  direction  to  the  test  performance;  and  to  ensure  that  the  test  was  conducted 
smoothly  and  on  schedule.  The  Test  Director  was  supported  by  two  data  collectors  from 
Essex  Corporation,  one  computer  specialist  from  Hughes  Aircraft  Corporation,  and  a 
video  camera  crew  from  DTRC.  The  data  collectors  recorded  information  on  subjects' 
performance  time  and  on  the  number  and  type  of  errors  committed  during  performance  of 
the  test  tasks,  and  conducted  the  debriefings.  The  Computer  Specialist  briefed  the 
subjects  on  the  electronic  delivery  device  used  and  on  the  NTIPS  TI  and  assisted  with  the 
delivery  device  operation  as  needed  throighout  the  test.  The  video-camera  crew  taped 
the  maintenance-task  performance  of  several  subjects  and  also  taped  all  debriefings.  In 
addition,  interviews  were  taped  with  instructor  personnel  who  participated  in  the  field 
test.  During  these  interviews,  instructors  provided  their  own  assessments  of  the  NTIPS 
TI,  of  the  test,  and  of  the  subjects'  performances  during  the  test. 
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TkUe  1.  Test  pen(»inel  and  functions. 


Source  of  Test  Personnel 


Function 


NAS  Miramar  (VF-124  FRAMP) 


Mane^ement  Personnel 
(Senior  Enlisted 
Instructor} 

Instructor  Personnel 
(Senior  Enlisted 
Instructors) 


0  Coordinated  scheduling  of  subjects  and 
facilities 


Delivered  test  briefings 
Monitored  subject-technician  per¬ 
formance 

Provided  assistance  to  subject- 
technician  in  performing  multiperson 
tasks 

Provided  evaluation  of  subject- 
technician  performance 


Subject  Technicians  (24) 
(AEs  experienced  and 
inexperienced) 


DTRC 


Performed  troubleshooting  and  correc¬ 
tive  maintenance  tasks  in  accordance 
with  conventional  paper  TI  and/or 
NTIPS  electronic/paper  TI 
Responded  to  preference  questionnaire 
and  post-test  briefing  (evaluation) 


Test  Director 


Video  Camera  Crew 


Served  as  principal  NTIPS 
representative 
Directed  the  test 

Coordinated  test  with  NAS  Miramar 
personnel:  scheduling,  obtaining  test 
subjects,  and  handling  test  logistics 

Videotaped  maintenance  task 
performance 

Videotaped  subjects'  debriefings 
Videotaped  interviews  with 
participating  FRAMP  instructors 
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Tilda  1  (CootlmiedO 


Source  of  Test  Personnel  Function 


TTiicrhM  Aircraft  Comoration  o  Set  up  and  checked  out  NTIPS  H  in 

both  the  paper  and  the  electronic 
delivery  media 

o  Briefed  teat  subjects  on  NTIPS  TI  and 
on  operation  of  the  electronic  delivery 
device 

o  Assisted  in  the  operation  of  the 
electronic  delivery  device,  as  needed 

Essex  Comoration  o  Collected  data  on  technician 

-  activities,  performance  times,  and 

errors 

o  Conducted  debriefings  of  subjects  and 
administered  preference 
questionnaires 

o  Analyzed  data  and  prepared  test 
report 
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3.2  MAINTENANCE  TASK  DEFINinON 

Test  subjects  performed  both  troubleshooting  (fault  isolation)  and  corrective- 
maintenance  tasks  on  the  F-14A  Rudder  Manual  Trim  System  (RMTS).  The  RMTS  is  part 
of  the  Direction  Control  System  of  the  F-14A  aircraft.  Specifically,  the  RMTS  affects 
the  yaw-control  function  by  trimming  the  rudders  to  compensate  for  asymmetric  flight 
about  the  longitudinal  axis  of  the  aircraft.  The  major  components  of  the  RMTS  are  the 
Pushrods  (PR),  the  Directionad  Trim  Actuator  (DTA),  and  the  Rudder-Surface  Position 
Indicator  (RSPl).  The  troubleshooting  problem  selected  for  the  field  test  was  to  find  a 
loose  ground  wire  to  the  DTA.  When  the  ground  was  disconnected,  the  DTA  could  not  be 
activated  by  the  rudder  trim  switch  and  the  rudders  did  not  move.  This  fault  was  inserted 
by  a  FRAMP  instructor  at  the  beginning  of  each  test  run.  To  start  the  test  sequence,  the 
subject  was  given  a  VIDS/MAF  form,  prepared  by  the  workshop  chief,  indicating  that  the 
rudder  trim  was  inoperable.  The  subject  then  proceeded  to  verify  the  fault  and  to 
perform  appropriate  tests  to  isolate  the  faulty  component. 

The  corrective-maintenance  tasks  selected  for  the  test  included  the  removal  and 
reinstallation  of  the  DTA,  zeroing  and  attaching  the  rudder  protractor,  and  checking  and 
adjusting  the  rudders.  The  following  criteria  were  used  in  selecting  these  tasks: 

o  Tasks  must  be  capable  of  being  performed  in  an  operational  environ¬ 
ment  represented  by  the  maintenance  facilities  available  at  a  Naval 
Air  Station,  with  no  augmentation  of  any  kind. 

o  Tasks  must  be  presented  to  the  technician  in  a  way  that  conforms  to 
procedures  and  methods  normally  used  in  an  operational  setting. 

o  Tasks  must  be  capable  of  being  performed  by  the  type  of  technicians 
called  for  and  available  in  typical  operational  maintenance  activities. 

o  Performance  of  the  task  must  not  require  support  effort  (e.g., 
rigging)  for  which  NTIPS  TI  is  not  available,  nor  should  it  require 
procedures  which  could  have  an  adverse  effect  on  the  operational 
availability  of  the  aircraft  (e.g.,  interruption  of  a  hydraulic  line). 

o  A  task  should  be  neither  too  simple  nor  too  complex.  The  former 
would  have  provided  inconclusive  results,  and  the  latter  would  have 
required  too  lengthy  an  overall  test  schedule. 

Both  the  troubleshooting  and  the  corrective-maintenance  tasks  selected  for  the 
tests  were  representative  of  maintenance  tasks  regularly  performed  on  the  F-14A.  Table 
2  identifies  each  test  task  selected  and  shows  its  position  in  a  typical  maintenance 
sequence  carried  out  at  a  Naval  Air  Station. 
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TAKLB  2.  Identifieation  and  definition  of  troubleebooting  and 
eoweetive'  meintenence  tasks. 


Task  Title 


1.  Ready  Aircraft  tor  Trouble¬ 
shooting  and  Corrective  Main¬ 
tenance 


2.  Verify  the  Fault 

-  Check  Rudder  Operation  with 
Pedals 

-  Check  Rudder  Operation  with 
Directional  Trim  Switch 


3.  Troubleshoot  to  Isolate  Faulty 
Component 

-  Locate  Plug  Connected  to 
DTA 

-  Test  Appropriate  Pins 


4.  Remove  Faulty  Component 


Task  Definitions 


Before  any  maintenance  work  can  be 
done,  certain  safety,  power,  and  system 
conditions  must  be  set.  Thhs  task 
establishes  these  conditions. 

A  (VIDS/MAF)  form  is  used  to  indicate 
to  the  technician  which  system  or 
subsystem  may  be  causing  problems. 

The  technician  selects  the  relevant  TI 
and  follows  its  instructions  to  verify 
that  the  reported  malfunction  does  in 
fact  exist.  (In  this  test,  a  fault  was 
inserted  in  an  operational  aircraft  by  a 
FRAMP  instructor  to  ensure  a  realistic 
procedure.)  Fault  symptoms  resulting 
from  fault  verification  serve  as  the 
basis  for  entering  the  FIND  automated 
troubleshooting  system  or  the 
troubleshooting  part  of  the 
conventional  Technical  Manual. 

In  this  task,  the  technician  follows  the 
troubleshooting  TI  to  identify  the 
component  causing  the  fault  symptom; 
i.e.,  to  perform  fault  isolation.  For  the 
NTIPS  TI,  these  step-bystep 
troubleshooting  instructions  are  called 
FIND  (Fault  Isolation  by  Nodal 
Dependency).  The  technician  obtains 
these  instructions  by  interacting  with 
the  NTIPS  electronic  delivery  device. 

The  technician  begins  the  process  of 
correcting  the  malfunction  by  removing 
the  component  which  his  testing  has 
identified  as  faulty. 
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Table  2  (C<mtinued) 


Task  Title 


Task  Definitions 


5.  Install  a  New  Component 


6.  Conduct  Operational  Check 

-  Attach  Rudder  Protractor 

-  Check  and  Adjust  Rudder 
Positioning 


7.  Restore  the  Aircraft  to 
Operational  Condition 


8.  Complete  Maintenance  Records 


After  obtaining  a  workii^  component 
from  Supply,  the  technician  installs  it 
in  the  system  in  place  of  the  faulty 
component  he  has  removed. 

The  technician  performs  an  operational 
check  of  the  aircraft  to  verify  that  the 
preceding  actions  (1)  have  eliminated 
the  malfunction  and  (2)  have  not 
introduced  a  new  fault  into  the 
aircraft. 

The  technician  restores  the  aircraft  to 
operational  readiness  by  eliminating 
conditions  which  were  changed  to 
permit  maintenance,  e.g..  Reset 
Switches.  Restoration  to  full 
operational  readiness  will  be  verified  by 
a  senior  instructor. 

The  technician  reports  the  completed 
work  on  the  appropriate  maintenance 
action  forms;  e.g.,  VIDS/MAF  form. 


Note;  Experimental  (NTIPS)  TI  and  conventional  TI  were  provided  for  all  eight  tasks.  To 
ensure  technician  and  aircraft  safety.  Task  1  was  performed  by  senior  FRAMP  personnel. 
Tasks  2  and  3  constituted  the  Troubleshooting  Tests.  Tasks  4-7  constituted  the 
Corrective-Maintenance  Tasks. 


3.S  TEST  PLANS  AND  SUBJBCTS 

The  test  was  designed  so  that  comparisons  could  be  made  on  a  technician's 
performance  using  each  type  of  TI.  Five  task/TI  combinations  were  compared  in  the  field 
test: 

o  Troubleshooting  using  FIND  (NTIPS  -  Electronic) 
o  Troubleshooting  using  conventional  paper  TI 
o  Corrective  maintenance  using  NTIPS  TI  in  electronic  form 
o  Corrective  maintenance  using  NTIPS  TI  in  paper  form 
o  Corrective  maintenance  using  conventional  paper  work-package  TM's. 

Each  subject  performed  the  troubleshooting  task  in  accordance  with  only  one  TI 
type»  and  the  corrective-maintenance  task  with  a  different  TI  type.  Subjects  using  FIND 
for  troubleshooting  used  conventional  paper  TI  or  NTIPS  paper  TI  for  corrective 
maintenance.  Subjects  using  conventional  paper  TM's  for  troubleshooting  used  NTIPS 
electronic  TI  or  NTIPS  paper  TI  for  corrective  maintenance.  Subjects  were  divided  into 
equal  groups,  with  one  half  performing  troubleshooting  with  each  type  of  troubleshooting 
TI  and  one  third  performing  corrective  maintenance  with  each  type  of  corrective- 
maintenance  TI.  Additionally,  subjects  were  assigned  to  two  groups  based  on  the  amount 
of  their  F-14A  experience;  each  of  these  contained  12  Aviation  Electrician's  Mates  (AEs). 

This  test  design  is  shown  in  Fig.  1,  which  is  reproduced  from  the  Test  Plan.  As  the 
test  progressed,  some  modifications  were  made  to  the  Test  Plan  based  on  the  availability 
of  subjects.  As  a  result,  the  field  test,  as  actually  performed,  involved  13  experienced 
technicians  and  11  inexperienced  technicians.  For  troubleshooting,  half  used  FIND  and 
half  used  conventional  paper  TI;  for  corrective  maintenance,  8  used  NTIPS  electronic  TI, 
10  used  NTIPS  paper  TI,  and  6  used  conventional  paper  TI.  Fig.  2  shows  the  actual  test 
configurations  used,  including  the  number  of  experience  (E)  and  inexperienced  (I) 
technicians  used  in  each  case. 

Subjects  were  classified  as  experienced  (E)  if  they  had  one  year  or  more  of  relevant 
F-14A  maintenance  experience  and  if  the  Maintenance  Chief  in  VF-124  FRAMP  judged 
that  this  experience  in  specific  cases  was  of  good  quality.  The  quality  of  this  F-14A 
experience  was  carefully  weighed  in  assigning  technicians  to  the  E  or  I  category.  In  two 
cases,  technicians  (both  E-4s)  with  slightly  more  than  a  year  of  F-14A  experience  were 
nevertheless  categorized  as  inexperienced,  and  one  E-8  (AE-1)  with  more  than  21  years  in 
the  Navy,  two  of  those  years  involvii^  administrative  aspects  of  F-14A  maintenance,  was 
classified  as  inexperienced.  No  technician  in  the  E  class  had  less  than  one  year  of  F-14A 
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Figure  1.  Test  plan  design. 
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(I.*.,  8E  MEANS  SIX  EXPERIENCED  TECHNICIANS). 


Figure  2.  Performance  of  test  tasks  by  technicians  during  field  test. 
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hands-on  maintenance  experience.  Inexperienced  technicians  (I)  were  included  in  the  test 
to  uncover  specific  problems  of  clarity  in  NTIPS  presentations  of  graphics  and  text. 
Table  3  lists  all  subjects  and  their  backgrounds.  Of  specific  interest  were: 

o  Rate 

0  Time  in  the  Navy 

o  Time  spent  performing  maintenance  on  the  F-14A 
o  Type  of  maintenance  experience 
o  Amount  of  computer  experience. 

A  technician's  computer  experience  was  considered  important  because  of  its  possible 
influence  on  his  acceptance  of  or  ability  to  achieve  familiarity  with  the  electronic 
delivery  device  and  on  his  assessment  of  its  usability  and  usefulness. 

3.4  DRY  RUN  AND  PRETEST  EVENTS 

3.4.1  The  Dry  Run 

Test  personnel  conducted  an  initial  walkthrough  of  the  test  procedures  on  2  May 
1986  under  the  observation  of  FRAMP  instructors.  The  following  test  parameters  were 
assessed: 

a.  Obtain  a  preliminary  evaluation,  by  senior  maintenance  personnel,  of  the 
experimental  TI  in  terms  of  test  suitability. 

b.  Work  out  the  interrelationships  between  test  subjects,  the  personnel 
recording  test  data,  and  the  FRAMP  instructor  serving  as  test  observer. 

c.  Establish  physical  locations  for  equipment,  aircraft,  delivery  devices, 
and  other  test-support  items. 

d.  Safety  considerations. 

A  number  of  changes  in  proposed  procedures  and  in  the  actual  TI  were  made  as  a  result  of 
the  dry  run. 

3.4.2  The  Pretest  Event 

During  the  period  9-11  September  1986,  two  AE-3s  and  two  AE-2s  performed  the 
maintenance  tasks  planned  for  the  test.  These  efforts  were  carefully  observed  by  test 
personnel  and  by  FRAMP  instructors  to  make  sure  that  the  refined  test  plan  was  optimsil 
for  (1)  efficient  use  of  test  subjects  and  observer  personnel,  (2)  safe  conduct  of  the  tests, 
and  (3)  achieving  the  test  objectives.  The  results  of  this  pretest  event  were  formally 
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Table  S.  Biographieel  data  for  teat  8«d)}eet8. 


Inei^perieneed  Subjects 


Time  in 

Time  with  Maintenance 

Computer 

Rate 

Naw 

F-14A 

Level 

Proficiency 

E-4 

2  yr,  1  mo 

1  yr,  4  mo 

0 

No 

E-6 

12  yr 

7  mo 

0 

Some 

E-3 

1  yr,  S  mo 

4  mo 

0 

No 

E-6 

21  yr,  6  mo 

2  yr* 

0 

No 

E-5 

3  yr,  1  mo 

6  mo 

0 

No 

E-3 

1  yr,  10  mo 

10  mo 

O 

Some 

E-4 

2  yr,  2  mo 

1  yr,  2  mo 

0 

No 

E-3 

1  yr,  10  mo 

7  mo 

0 

Some 

E-3 

1  yr,  10  mo 

8  mo 

0 

Yes 

E-4 

1  yr,  10  mo 

7  mo 

0 

No 

E-4 

1  yr,  10  mo 

3  mo 

O 

No 

Experienced  Subjects 

E-4 

3  yr,  6  mo 

2  yr,  6  mo 

0 

No 

E-4 

2  yr,  5  mo 

1  yr,  3  mo 

0 

Some 

E-5 

5  yr,  4  mo 

1  yr 

0 

No 

E-4 

3  yr,  6  mo 

2  yr,  6  mo 

0 

No 

E-4 

4  yr 

3  yr,  9  mo 

0 

Yes 

E-4 

2  yr 

1  yr,  6  mo 

0 

Yes 

E-5 

7  yr,  8  mo 

4  yr,  6  mo 

0 

No 

E-6 

7  yr 

2  yr 

0 

Yes 

E-5 

7  yr,  8  mo 

7  yr 

0 

No 

E-5 

8  yr,  9  mo 

1  yr,  10  mo 

O 

No 

E-5 

6  yr,  4  mo 

5  3rr,  4  mo 

I/O 

Yes 

E-4 

3  yr,  8  mo 

2  yr,  7  mo 

0 

No 

E-4 

2  yr,  10  mo 

1  yr,  9  mo 

0 

No. 

All  technicians  were  AEs.  Thus,  E-3  corresponds  to  AEAN;  E''4  to  AE-3;  £-5  to  AE-2;  and 
E-8  to  AB-1. 

Participating  Organizations:  NAMTRADET,  AIMD,  VF-124,  VF“114,  VF-24,  VF-211,  VF- 
213,  VF-301,  VP-1,  VF-154,  VF-2,  VF-21 


O  -  Organizational,  I  -  Intermediate 
*  Paperwork  in  shop. 


documented  in  an  NTIPS  Report:  "Navy  Technical  Information  Presentation  Program. 
Test  and  Evaluation  of  the  Navy  Technical  Information  Presentation  Systems  (NTIPS).  F- 
14A  Initial  Evaluation",  by  Anne  S.  Mavc^  and  Theodore  J.  Post,  Essex  Corporation, 
September  1986. 

The  Pretest  event  (and  the  previous  Dry  Run)  thus  constituted  a  verification  of  the 
experimental  Technical  Information,  a  vital  process  in  the  generation  of  all  system- 
related  TI  required  to  ensure  the  operational  suitability  (consistency  with  fleet  proce¬ 
dures,  technician  capability,  and  operational  environment)  of  the  TI. 

As  a  residt  of  the  Pretest  event, 

1.  Some  final  changes  were  made  to  the  experimental  TI 

2.  Logistic  arrangements  for  test  conduct  were  finalized 

3.  Final  changes  to  the  test  procedure  were  incorporated  to  increase  test 
efficiency. 

For  example,  during  the  pretest  event,  it  was  observed  that  time  spent  in 
performance  of  two  of  the  five  Corrective-Maintenance  Tasks  (specifically  4  and  5)  were 
far  more  dependent  on  the  manual  dexterity  of  the  test  subjects  than  on  the  nature  of  the 
TI  used,  since  both  of  these  tasks  (remove  a  Directional  Trim  Actuator  and  replace  a 
Directional  Trim  Actuator)  involved  connecting  and  disconnecting  parts  in  a  relatively 
inaccessible  area.  Consequently  the  procedure  was  modified  so  that  the  test  subject  used 
the  TI  for  these  tasks  but  orally  reported  to  the  observers  what  he  would  do  to  the 
equipment,  e.g.,  touched  the  part  he  believed  to  be  involved  and  orally  described  how  he 
would  perform  the  step.  Performance  of  Corrective  Maintenance  Tasks  6  through  8  was 
carried  out  as  originally  planned. 


3.5  TEST  SCHEDULE  AND  PERFORMANCE 

The  field  test  was  conducted  during  the  period  from  October  14  to  October  24,  1986. 
Two  or  three  subjects  performed  the  test  tasks  each  day.  The  schedule  of  events  for  each 
subject  was  as  follows: 

o  Attend  a  test  briefing  presented  by  an  NAS,  Miramar,  test  coor¬ 
dinator.  This  briefing  covered  the  test  purpose,  the  tasks  to  be 
performed,  the  estimated  time  required,  and  the  importance  to  the 
program  of  filling  out  the  preference  questionnaire  and  participating 
in  the  debriefing. 

o  Attend  a  technical  briefii^  presented  by  a  Hughes  Aircraft  represen¬ 
tative.  This  briefing  covered  FIND  and  the  electronic  and  paper 
versions  of  NTIPS  for  corrective  maintenance.  As  part  of  this 


briefing  the  subject  was  given  hands-on  experience  with  the  TI,  the 
display,  the  keytMard,  and  the  touch  panel. 

0  Receive  a  VIDS/MAF  form  (problem  report)  leading  to  the  perform¬ 
ance  of  fault  verification  and  troubleshooting  using  either  FIND  or 
conventional  paper  TL* 

o  Perform  corrective  maintenance  with  one  of  the  following  TI  types: 

NTIPS  electronic  delivery,  NTIPS  paper,  or  conventional  paper  work 
package  TM. 

o  Fill  out  a  questionnaire  designed  to  obtain  evaluative  comments  from 
the  test  subjects  concerning  the  use  of  NTIPS  TI,  delivered  by  paper 
or  electronically,  and  traditional  paper  TI. 

o  Participate  in  a  debriefing  interview  to  evaluate  the  various  TI  forms 
and  media  used  and  their  relative  effect  on  the  speed  and  accuracy  of 
task  performance.  Provide  opinions  on  new  TI  approaches. 

3.6  THE  ELECTRONIC  DELIVERY  DEVICE 

The  electronic  delivery  device  was  an  AT&T  3B-2/300  with  an  AT&T  6300  touch 
screen  and  keyboard.  A  touch  screen  works  by  recording  a  signal  when  a  finger  interrupts 
infrared  beams  which  form  a  matrix  across  the  front  of  the  screen.  When  a  pair  of  beams 
is  broken  by  a  finger  passing  through  the  matrix,  the  corresponding  point  on  the  screen  is 
identified  as  being  "touched."  The  computer  terminal  was  hooked  up  to  a  printer  and  to  a 
display  repeater  terminal  (on  the  deck)  that  showed  the  same  image  as  the  primary  display 
screen. 

During  the  test,  the  computer  (delivery  device)  was  located  on  an  elevated  test 
stand  at  the  wing  root  of  the  F-14A.  The  aircraft's  wings  were  swept  forward.  When 
using  the  TI  delivery  device,  the  technician  stood  on  top  of  the  aircraft  in  front  of  the 
device,  reviewed  some  subset  of  the  TI,  and  then  moved  to  one  of  the  work  sites  (also  on 
top  of  the  aircraft).  Depending  on  the  subject’s  approach  to  the  task  and  the  TI,  several 
trips  between  the  computer  and  the  work  site  could  be  required.  These  work  sites  atop 
the  aircraft  were  the  cockpit,  the  rudder  area,  and  the  access  opening  to  the  DTA. 
Distances  from  the  delivery  device  were  approximately  30  feet,  15  feet,  and  10  feet, 


•  Conventional  work-package  troubleshooting  TI  contains  both  logic-tree  procedures  and 
schematic  diagrams  showing  the  component  relationship  and  signal  flow.  The  work 
package  did  not  contain  logic  tree  procedures  relevant  to  the  test  fault.  The  relevant 
schematics  for  the  fault  were  contained  in  NAVAIR  01-F14AAA-2-2-16-3,  Technical 
Manual,  Organizational  Maintenance,  Integrated  Weapon  Systems  Functional  Diagrams, 
Navy  Model  F-14A  Aircraft,  Change  3,  15  June  1985,  Fig.  47.  At  Miramar  NAS,  the 
invariable  practice  is  to  perform  fault  isolation  with  the  schematics  rather  than  the 
logic-tree  diagrams. 
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respectively.  Any  screen  image  could  be  printed  by  executing  a  print  command.  This 
capability  allowed  a  subject  to  make  a  hard  copy  and  take  it  with  him  to  the  work  site. 

The  repeater  screen  was  located  on  the  deck  and  could  be  used  by  the  subject  in 
performing  those  steps  in  the  maintenance  procedxire  which  required  performance  on  the 
deck.  Distances  from  the  repeater  to  the  two  work  sites  involved  were  both  approxi¬ 
mately  15  feet.  (Note;  If  electronic  delivery  is  approved  for  implementation  in  the  Navy, 
a  portable  device  would  be  used  to  take  to  the  equipment  to  be  maintained). 

3.7  DATA  COLLECTION  CATEGORIES  AND  PROCEDURES 

Data  collection  categories  included  (a)  two  performance  measures  (performance 
time  and  errors  committed  during  task  performance),  (b)  one  descriptive  measure  (actions 
engaged  in  by  technicians  during  task  performance),  (c)  technician's  subjective  ratings 
(poor  to  excellent)  of  the  quality  of  the  text  and  graphics  composing  the  TI  they  used,  and 
(d)  technician's  preferences  for  electronic  or  paper  presentation.  Sample  data-collection 
forms  for  corrective-maintenance  task  performance  are  provided  in  Appendix  A  (the 
forms  used  for  troubleshooting  performance  evaluation  were  essentially  identical).  The 
preference  questionnaire  is  provided  in  Appendix  B.  One  data  collection  form  was  used  to 
record  the  actions  engaged  in  by  the  subject  when  performing  each  step  in  the  procedure 
and  the  time  occupied  by  each  type  of  action.  The  other  form  was  used  to  record  and 
describe  any  errors  made  by  the  subject.  Each  of  these  forms  listed  the  task  steps  to  be 
performed. 

During  the  test,  the  subjects'  actions  during  the  prosecution  of  the  test  tasks  were 
codified  as  follows; 

o  TI  —  Using  Technical  Information 

o  C  —  Communicating  with  helper 

o  W  —  Performing  work  on  the  system 

o  TI/C  —  Perusing  TI  and  communicating  at  the  same  time  (back  and 
forth  between  perusing  TI  and  talking  to  helper) 

o  TI/W  —  Perusing  TI  and  working  (actually  moving  or  changing  parts) 
at  the  same  time  —  back  and  forth  between  TI  and  work 

o  C/W  —  Communicating  and  working  at  the  same  time  (back  and  forth 
between  work  and  communicating  with  helper) 

o  TI/E  —  Perusing  TI  and  examining  equipment  (back  and  forth 
between  TI  and  looking  at  equipment). 

These  categories  were  used  in  monitoring  field  test  performance  to  provide  a  record 
showing  what  the  subject  did  to  complete  each  maintenance  step.  The  procedure  used  was 
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to  record  each  action  code  while  the  corresponding  action  was  occurring  and  to  record  the 
elapsed  time  until  the  subject  went  on  to  another  type  of  action.  Timing  was  initiated  at 
the  beginning  of  each  subtask.  Thus,  within  each  subtask  or  step,  it  was  possible  to 
determine  the  amount  of  time  spent  working,  communicating,  examining,  perusing  TI,  etc. 

The  errors  made  by  a  subject  were  recorded  on  a  second  data  collection  form. 
Categories  of  errors  included 

o  False  Starts:  Begins  to  work  on  equipment,  stops,  looks  back  at  TI, 
and  then  starts  over. 

o  Wrong  Location:  The  subject  attempts  to  find  a  component  in  a 
different  aircraft  location  from  the  location  listed  in  the  step 
description. 

o  Wrong  Part:  The  subject  uses  the  wrong  part  or  tool. 

o  Helpers:  The  subject  uses  helpers  when  they  are  not  appropriate  or 
does  not  use  helpers  when  they  are  appropriate. 

o  Other  Errors:  Errors  not  covered  by  the  above  categories. 

Measures  of  performance  time  and  accuracy  were  analyzed  by  subtask,  by  type  of  TI 
used,  and  by  subject  experience.  The  outcomes  of  these  analyses  are  reported  in  the 
Results  section  (4.0).  In  addition,  a  post-test  debrief  of  the  test  subjects  was  conducted. 
This  debrief  was  used  to  determine  the  subjects'  attitudes  (e.g.,  acceptance  or  dislike) 
toward  individual  types  of  TI  and  the  presentation  media  used.  By  means  of  a  written 
questionnaire,  subjects  were  asked  to  express  their  reactions,  for  or  against,  specific  TI 
characteristics,  including  style,  content,  format,  delivery  medium,  and,  in  general,  the 
ease  of  use  of  the  TI.  It  is  important  to  assess  which  TI  characteristics  caused  the 
intended  users  to  view  TI  as  beneficial  and  easy  to  use,  or  —  on  the  other  hand 
troublesome  and  confusing.  A  negative  reaction  in  this  category  of  data  might  indicate 
areas  in  which  improvement  could  be  required  before  such  a  system  could  be  introduced 
into  the  Navy.  The  questionnaire  used  to  obtain  the  users'  reactions  appears  in  Appendix 
B.  This  questionnaire  includes  a  scale  to  allow  the  respondents  to  report  the  intensity  of 
their  reactions,  negative  or  positive,  on  a  scale  of  1  to  5  to  the  individual  TI 
characteristics.  The  first  sheet  of  the  questionnaire  presents  the  rating  scale  and 
provides  instructions  for  its  use.  The  remaining  sheets  list  the  TI  characteristics  and 
provide  spaces  for  the  subjects  to  mark  their  responses.  The  questionnaire  contains  four 
sections:  Part  I  deals  with  Technical  Information,  Part  II  with  electronic  delivery,  Part  III 
with  comparing  electronic  delivery  and  the  conventional  paper  Technical  Information,  and 
Part  IV  with  biographical  data. 
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In  addition,  each  subject  was  interviewed  to  allow  him  to  express  any  opinions  not 
covered  by  the  questionnaire.  An  outline  of  this  interview  appears  as  the  last  page  of 
Appendix  B. 
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4.0  RESULTS 


4.1  SUMMARY  OF  RESULTS 

The  objectives  of  the  NTIPS  field  test  were  as  follows: 

o  Compare  the  performance  of  enlisted  maintenance  technicians  using 
TI  prepared  under  NTIPS  procedures  to  the  performance  of  techni¬ 
cians  using  conventional  TI  (the  paper  F-14A  Technical  Manual). 

o  Compare  technicians'  performance  when  using  (NTIPS)  TI  printed  on 
paper  to  performance  using  (NTIPS)  TI  presented  via  an  electronic 
display  device. 

o  Establish  which  design  characteristics  of  NTIPS  TI  are  most  effective 
and  least  effective  in  an  operational  situation. 

o  Assess  user  acceptance  of  the  NTIPS  modes  (medium,  content, 
format,  and  style)  of  TI  presentation. 

In  addition,  the  field  test  was  designed  to  provide  an  evaluation  of  the  NTIPS  TI  and 
various  aspects  of  the  delivery  device  in  such  a  way  as  to  indicate  the  need  for 
improvements  or  modification  in  either  the  TI  or  in  the  display  system;  i.e.,  it  was 
designed  to  provide  "a  formative  evaluation."  The  data  on  subject  preference  for  various 
aspects  of  the  NTIPS  TI  and  the  use  of  electronic  delivery  of  TI  have  been  analyzed,  and 
the  results  are  reported  below.  The  general  findings  were  that  most  (90%)  of  the  subjects 
preferred  electronic  delivery  of  TI  over  conventional-paper  TI  presentation  and  they  were 
able  to  use  NTIPS  TI  to  perform  more  accurately  and,  in  some  cases,  with  greater  speed 
than  when  conventional  paper  TI  was  used.  A  summary  of  the  favorable  comments  about 
NTIPS  is  as  follows: 

0  Electronic  delivery  provides  easier  access  to  desired  sections  of  the 
TI;  the  technician  does  not  need  to  look  through  several  volumes  of 
paper.  Access  can  be  achieved  by  one  or  two  keystrokes. 

o  FIND  was  easy  to  follow  and  would  be  most  useful  for  troubleshooting 
complex  or  unfamiliar  systems. 

o  Electronic  TI  saves  space  and  is  easier  to  update  than  conventional 
paper. 

A  more  detailed  review  of  these  comments  is  provided  in  the  section  on  Preference  Data 
(4.4). 
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4.2  PERFORMANCE  TIMES  AND  ACTIONS 

As  noted,  several  classifications  were  established  for  collecting  data  describing 
subjects'  actions  during  the  task  performance;  however,  two  categories,  T1  perusal  (Tl) 
and  work  performance  (W),  accounted  for  most  of  the  performance  time.  On  the  basis  of 
this  finding,  times  spent  on  all  other  action  categories  (e.g.,  communicating,  examining 
the  equipment,  etc.)  were  subsequently  combined  and  labeled  "miscellaneous  (M)"  for 
purposes  of  analysis.  Analyses  of  the  performance  times  were  conducted  for 
troubleshooting  and  corrective  maintenance. 

4.2.1  Troubleshooting  Performance  Times 

Troubleshooting  was  divided  into  two  tasks:  fault  verification  and  fault  isolation. 
Half  of  the  24  subjects  performed  these  tasks  with  FIND,  while  the  other  half  used 
conventional  paper  Technical  Manuals.  For  the  selected  troubleshooting  task,  the 
conventional  paper  TM  consisted  only  of  schematics.  (No  troubleshooting  trees  relevant 
to  the  test  fault  were  included  in  the  conventional  TM.)  When  the  NTIPS  electronic 
presentation  was  used,  the  fault-verification  procedure  was  directed  by  the  display  device 
as  a  result  of  inserting  specific  trouble  symptoms  and  test  results.  This  was  not  the  case 
with  the  conventional  paper  TM.  As  a  result,  some  of  the  subjects  using  the  conventional 
TM  went  directly  into  fault  isolation,  without  performing  the  fault-verification  procedure. 
The  FIND  sequence  of  computer-presented  questions  and  technician-presented  answers 
not  only  verifies  the  reported  problem,  but  provides  the  basis  for  initiating  the  fault- 
isolation  procedure. 

Table  4  compares  the  performance  times  for  experienced  subjects  using  NTIPS 
electronically  displayed  TI  (FIND)  with  those  using  conventional  paper  TM  for  trouble¬ 
shooting.  Comparison  of  the  total  task  performance  times  in  each  case  shows  that 
subjects  using  NTIPS  FIND  took  an  average  of  10  minutes  (37%)  longer  to  find  the  fault 
than  subjects  using  conventional  schematics.  When  these  totals  are  broken  down  into 
separate  activities  of  TI  perusal,  work,  and  miscellaneous,  the  following  pattern  emerges: 

0  For  NTIPS  FIND,  53%  of  the  total  work  time  is  spent  in  TI  perusal 
(TI) 

o  For  conventional  TI,  59%  of  the  total  time  is  spent  in  Work 

0  Subjects  using  FIND  spent  somewhat  less  time  working  (W)  (12.3 
minutes)  than  subjects  using  conventional  TM  (17.3  minutes). 


As  noted  in  Section  3.5,  some  troubleshooting  decision  trees  are  available  in  the 
conventional  TI  work  packages.  In  this  test,  of  the  12  test  technicians  who  performed 
fault  isolation  with  conventional  paper  TI,  two  (one  experienced  and  one  inexperienced) 
started  their  performance  by  looking  for  the  logic-tree  diagprams  as  the  basis  for 
troubleshooting.  (The  other  ten  began  with  the  schematics.)  Not  being  able  to  find 
relevant  logic  diagrams,  the  two  test  technicians  were  directed  by  the  senior  enlisted 
observers  to  the  schematics,  which  they  then  used  to  proceed  with  fault  isolation.  The 
experienced  technician  located  the  fault  with  the  schematics.  The  inexperienced 
technician  did  not  (nor  did  any  inexperienced  technician  using  conventional  TI).  As  a 
result  of  these  incidents,  test  performance  times  for  the  involved  technicians  were 
increased  by  7  minutes  and  14  minutes,  respectively.  If  these  times  were  removed  from 
the  averages,  the  new  total  time  for  experienced,  conventional  TI  would  be  28.1  minutes 
(Table  4);  for  inexperienced,  conventional  TI,  the  time  would  be  44.1  minutes  (Table  5). 


Table  4.  Troubleshooting  performance  times:  e^erienced  subjects. 

(times  in  minutes) 


TI  Perusal  (TI) 

Work  (W) 

Misc. 

Total 

FIND 

21.0 

12.3 

6.7 

40.0 

Conventional  TI 

8.8 

17.3 

3.2 

29.3 

These  results  suggest  that  FIND  made  it  possible  for  subjects  to  perform  the  work  faster; 
on  the  other  hand,  the  time  spent  perusing  the  TI  was  2.4  times  longer  for  FIND  than  for 
conventional  TI.  Observations  of  subjects  using  FIND  indicated  that  much  of  the  time 
spent  using  the  TI  was  occupied  in  responding  to  the  computer-based  questions  involved  in 
fault  verification,  waiting  for  images  to  change,  waiting  for  the  machine  to  respond  to 
test  results  provided  by  the  technician,  and  proceeding  through  the  detailed  instructions 
which  were  provided  for  each  test  point.  Observations  made  during  the  performances 
indicated  a  number  of  ways  in  which  FIND  approaches  can  be  streamlined.  When  this  is 
accomplished,  the  total  time  for  troubleshooting  with  the  electronic  delivery  device 
should  be  significantly  reduced. 

The  mean  performance  times  for  inexperienced  subjects  using  FIND  and  for  those 
using  conventional  TM  are  shown  in  Table  5.  The  results  show  that,  for  these  subjects. 
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FIND  shortened  overall  performance  time  somewhat  (8.1%).  That  is,  subjects  using 
conventional  TI  for  troubleshooting  took  loiter  than  subjects  using  FIND.  However,  none 
t  of  the  inexperienced  subjects  using  the  conventional  TI  found  the  fault  without  some 

assistance  from  the  instructor  who  was  serving  as  test  coordinator;  aU  subjects  usit^ 
FIND  located  the  fault  without  instructor  assistance  (see  Section  4.3.1).  A  comparison  of 
time  spent  on  work  and  TI  perusal  for  inexperienced  subjects  shows  more  time  spent 
perusing  the  TI  and  less  time  doing  work  when  FIND  is  used.  This  finding  is  similar  to  the 
pattern  shown  for  the  experienced  subjects.  However,  the  total  time  spent  on  trouble¬ 
shooting  using  the  two  types  of  TI  was  very  nearly  equal. 
i»  These  test  results  show  that  experienced  technicians  using  conventional  trouble- 

[  shooting  TMs  spend  a  relatively  small  portion  of  their  troubleshooting  time  perusing  the 

>  TI,  but  that  a  proceduralized  approach  like  that  of  FIND  forces  them  to  go  through  the 

entire  TI  process  and  thus  increases  the  TI  perusal  time  significantly  (a  factor  of  2.4). 
(For  inexperienced  technicians,  there  is  little  difference  in  the  perusal  times  for  the  two 
^  types  of  TI.)  To  what  extent  greater  familiarity  with  FIND  on  the  part  of  the  experienced 

technicians  and  accomplishment  of  clearly  indicated  improvements  in  the  FIND 
presentation  process  will  reduce  this  TI  perusal  time  remains  to  be  seen.  Further 
evaluation  is  required  to  assess  the  significance  of  the  result  that  work  time  was  reduced 
by  use  of  FIND  for  both  experienced  (28.8%)  and  inexperienced  (36.6%)  technicians. 

Table  S.  Troubleshooting  performance  times:  inexperienced  subjects. 

(times  in  minutes) 


TI  Use 

Work 

Misc. 

Total 

FIND 

22.4 

12.8 

7.7 

42.9 

Conventional  TI 

20.5 

20.1 

6.1 

46.7 

In  a  further  analysis  of  the  troubleshooting  performance  data,  work  activities  were 
broken  down  into  specific  steps.  After  a  review  of  the  TI,  the  first  step  was  to  remove 
the  access  cover  so  that  work  could  be  performed  on  the  DTA.  This  step  was  followed  by 
finding  the  plug  and  then  by  testing  voltages  on  the  pins.  When  conventional  TI  was  used, 
the  test  subject  generally  reviewed  the  appropriate  schematic,  noted  on  a  scrap  of  paper 
the  test  points  he  believed  to  be  relevant  ,  and  then  began  the  work.  With  FIND,  the 
subject  accessed  the  electronically  presented  TI  to  obtain  the  procedures  for  gaining 
access  to  the  plug,  testing  the  pins,  and  interpreting  the  test  outcomes. 

31 


Figure  3  shows  the  time  used  by  experienced  and  inexperienced  subjects  to  get  to 
each  point  in  the  task  sequence.  It  can  be  seen  that  fault  verification  always  takes  longer 
with  FIND  than  with  conventional  TI,  because  the  electronic  system  proceeds  in  a 
methodical  manner,  presenting  one  question  at  a  time  and  requiring  an  answer  before  the 
technician  can  move  on  to  the  next  question.  Time  is  required  to  move  from  one  image  to 
the  next,  and  in  some  cases  the  subjects  became  confused  as  to  where  they  were  in  the 
fault-verification  process.  For  the  test  case,  16  questions  were  posed.  The  answers  to 
these  questions  provided  the  initial  input  (trouble  symptoms)  to  FIND. 

In  the  fault-isolation  portion  of  the  task  process,  all  subjects  removed  the  cover  and 
found  the  connector  plug  faster  when  using  FIND  than  when  using  conventional  paper  (for 
experienced  technicians  12  vs.  18  minutes;  for  inexperienced  technicians  15  vs.  32 
minutes).  FIND  presents  detailed  instructions  with  graphics  showing  the  location  of  the 
access  cover  and  the  DTA.  Several  of  the  subjects  using  conventional  TI  had  -lifficulty 
locating  the  DTA  and  the  connector  plug,  particularly  the  inexperienced  subjects. 

Testing  time  with  FIND  took  significantly  longer  than  the  testing  time  with 
conventional  TMs.  Specifically,  experienced  subjects  took  16  minutes  to  perform  tests 
with  FIND  and  7  minutes  with  conventional  TM;  inexperienced  subjects  took  15  minutes 
and  8  minutes,  respectively.  Reducing  the  time  required  to  interact  with  the  FIND 
program  would  make  the  relative  times  required  more  nearly  equal. 

Finally,  all  subjects  using  FIND  were  able  to  isolate  the  fault  successfully,  whereas, 
30  percent  of  the  experienced  subjects  and  none  of  the  inexperienced  subjects  found  the 
fault  without  assistance. 

4.2.2  Corrective-Maintenance  Performance  Times 

Corrective-maintenance  consisted  of  the  following  tasks: 

o  Remove  DTA  (locate  component  and  describe  removal  steps) 
o  Install  DTA  (describe  installation  steps) 
o  Zero  Rudder  Protractor  — perform 
0  Attach  Rudder  Protractor  —  perform 
o  Check  and  adjust  rudder  positioning  —  perform 

Data  on  overall  performance  time  and  time  spent  in  each  type  of  action  were  summarized 
for  each  subject  by  task.  A  post-test  review  of  these  data  suggested  that  action-time 
data  be  presented  in  three  categories  —  TI  perusual,  work,  and  miscellaneous.  Further, 
the  individual  performance-time  results  were  averaged  across  tasks  and  across  subjects. 
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Figure  3.  Troubleshooting  steps  and  elapsed  time 


The  results  are  shown  in  Table  6.  This  table  compares  mean  performance  time  in  minutes 
for  experienced  and  inexperienced  subjects  using  each  of  the  three  types  of  TI.  There  is 
a  wide  range  in  the  time  spent  by  individual  subjects  in  performing  tasks  within  any  given 
category  of  TI  use.  As  a  specific  example,  one  inexperienced  subject  working  with  NTIPS 
electronic  took  three  times  as  long  as  any  other  subject  to  perform  Check  and  Adjust. 
This  technician's  recent  experience  involved  paper  work  rather  than  performing  mainte¬ 
nance  tasks.  His  performance  time  alone  is  responsible  for  the  increased  work  time  shown 
for  the  NTIPS  electronic-delivery  tests  (42.62  minutes)  for  inexperienced  subjects.  If  his 
time  is  removed,  the  average  work  time  in  this  test  condition  would  be  reduced  to  33 
minutes  —  more  in  line  with  that  obtained  using  the  other  types  of  TI. 

The  TI  perusal  time  for  both  experienced  and  inexperienced  subjects  is  slightly 
higher  for  NTIPS  electronic-delivery  TI  than  for  the  other  TI  categories.  Observations  of 
subjects*  behavior  at  the  work  site  suggest  two  reasons  for  this  result.  First,  the 
electronic  delivery  device  was  located  several  feet  from  the  worksite.  This  required  the 
subject  to  walk  back  and  forth  to  read  the  TI  and  then  perform  the  work.  In  many  cases 
the  subject  returned  to  the  display  two  or  more  times  to  check  his  memory  for  the  next 
step.  When  the  paper  TI  was  used,  the  subject  had  only  to  glance  down  quickly  to  find  his 
place.  (A  portable  TI  delivery  device,  of  the  type  contemplated  by  NTIPS  instead  of  the 
commercial  computer  used  in  the  tests,  would  of  course  eliminate  this  condition.)  The 
second  reason  was  that  the  graphics  on  the  electronic  display  were  not  as  clear  and 
detailed  as  the  graphics  in  both  types  of  paper  TI.  As  a  result,  subjects  spent  more  time 
studying  the  graphics  on  the  display.  This  was  particularly  true  for  the  steps  involved  in 
zeroing  and  attaching  the  rudder  protractor.  Almost  all  the  subjects  using  NTIPS 
electronic  TI  exhibited  confusion  when  using  the  graphics  to  identify  various  screws,  nuts, 
and  knobs.  The  tests  clearly  indicated  the  need  for  higher  quality  graphics  than  those 
incorporated  into  the  experimental  TI,  specifically  in  terms  of  fidelity  and  resolution. 
There  was  very  little  difference  in  the  amount  of  time  spent  perusing  the  TI  (in  any  of  the 
three  categories)  by  experienced  and  inexperienced  subjects. 

4.3  PERFORMANCE  ERRORS 

Performance  errors  were  analyzed  separately  for  troubleshooting  and  corrective 
maintenance.  The  most  significant  errors  in  troubleshooting  were  those  involving  a 
failure  to  isolate  the  fault.  In  corrective  maintenance,  there  were  no  failures  to  complete 
tasks,  but  there  were  some  incorrectly  performed  steps  and  inaccurate  identifications  of 
parts  or  part  locations.  These  errors  were  generally  corrected  by  the  subject  before 


TABLE  6.  Correetivnnaiiitenaiiee  performance  times. 
Mean  time  (minutes) 


Experienced  Subjects 


TI  Perusal 

Work 

Misc. 

Total 

Conventional  TI 

17.8 

32.3 

3.1 

53.2 

NTIPS  Paper 

19.1 

36.9 

6.5 

62.5 

NTIPS  Electronic 

22.7 

30.9 

6.1 

59.8 

Inexperienced  Subjects 

TI  Perusal 

Work 

Misc. 

Total 

Conventional  TI 

20.1 

26.7 

15.7 

62.5 

NTIPS  Paper 

17.1 

28.7 

13.4 

59.2 

NTIPS  Electronic 

25.8 

42.6* 

9.2 

77.6* 

*If  the  test  subject 

whose  experience  included  mostly  paperwork 

is  factored  out, 

the  work 

time  using  NTIPS 

electronic  is  reduced 

to  33  minutes  and  total  time  to  68 

minutes. 
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proceeding  to  the  next  step.  Specific  results  for  troubleshooting  and  corrective- 
maintenance  errors  are  discussed  in  the  following  sections. 

4.3.1  Troubleshooting  Error  Analysis 

The  errors  committed  by  subjects  while  performing  fault  verification  and  fault 
isolation  are  shown  in  Table  7.  Only  FIND  subjects  made  fault  verification  errors;  the 
absence  of  errors  for  this  task  when  conventional  paper  was  used  may  be  attributed  to  the 
lack  of  formal  fault  verification  procedures  in  the  conventional  TM.  Errors  with  FIND 
consisted  of  answering  the  fault  verification  questions  incorrectly.  These  errors  were  in 
fact  always  corrected  when  the  technicians  were  instructed  by  the  computer  to  verify 
their  responses. 

Table  7.  Troubleshootiiig  emm. 

FIND  Conventional  Paper 

Experienced  Inexperienced  Experienced  Inexperienced 

1  3 

1  4 

5 

1 

2  1  11 

2  5 

3  5  4  IS 


Fault  Verification 

Fault  Isolation 

-  Identified 
wrong  panel 

-  Identified 
wrong  plug 

-  Used  DC 
instead  of  AC 

-  Forgot  to  move 
trim  switch 

-  Failed  to 
isolate  the 
fault 

Totals 


For  fault  isolation,  most  of  the  errors  were  committed  by  inexperienced  subjects 
using  conventional  TI.  These  errors  were  in  two  categories:  identifying  the  wrong  access 
panel  and  identifying  the  wrong  connector  plug. 
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It  is  considered  important  that  all  of  the  inexperienced  subjects  and  28.6  percent 
(two  out  of  seven)  of  the  experienced  subjects  failed  to  find  the  fault  without  assistance 
when  using  the  conventional  TM.  All  subjects  using  FIND  isolated  the  fault  correctly 
without  instructor  assistance.  These  results  are  summarized  in  Table  8. 

Table  8.  Summary  of  fault-isolation  performance. 


FIND 

Subjects 

Success¬ 

fully 

Isolated 

Fault 

Number 

of 

Subjects 

Inexperienced 

6 

6 

Experienced 

6 

6 

Total 

12 

12 

Conventional  TM 


Percent 

Success¬ 

fully 

Number 

Percent 

Success¬ 

Isolated 

of 

Success¬ 

ful 

Fault 

Subjects 

ful 

100 

0 

5 

0 

100 

5 

7 

71.4 

100 

5 

12 

41.7 

4.3.2  Corrective-Maintenance  Error  Analysis 

Tables  9,  10,  and  11  show  the  distribution  of  errors  for  corrective-maintenance  tasks 
for  inexperienced  subjects,  experienced  subjects,  and  all  subjects,  respectively.  The 
results  for  inexperienced  subjects  (Table  9)  indicate  a  mean  of  8  errors  per  subject  for 
those  using  conventional  paper  TI,  as  compsmed  with  5.3  and  5.2  errors  per  subject, 
respectively,  for  NTIPS  electronic-delivery  and  NTIPS  paper  TI.  Thus,  for  this  test,  there 
was  a  35%  reduction  in  the  number  of  errors  in  corrective-maintenance  task  performance 
using  the  NTIPS  presentation  approaches.  The  small  number  of  subjects  in  each  group 
makes  it  difficult  to  generalize  conclusions  from  these  data  in  any  quantitative  sense. 
For  experienced  subjects  (Table  10),  there  were  no  significant  differences  in  results  for 
the  different  types  of  TI;  the  range  in  mean  performance  errors  per  subject  was  3.5  to  4.0. 
When  errors  were  combined  for  all  subjects  (Table  11),  the  range  in  mean  performance 
errors  per  subject  was  4.5  for  NTIPS  electronic  to  5.2  for  conventional  paper. 

Figure  4  shows  the  distribution  of  errors  for  each  TI  condition.  The  bars  represent 
the  percentage  of  total  errors  for  each  type  of  TI  attributable  to  a  given  error  category. 
The  major  findings  are  as  follows: 

0  The  largest  percentage  of  errors  for  NTIPS  electronic  and  NTIPS 
paper  TI  is  associated  with  performing  a  step  incorrectly. 
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TaUe  9.  Cotreetive-maintenaiiee  enoni  ineipaieiieed  sul^eets. 


NTIPS(E) 

NTIPSIP) 

CONV(PI 

Remove  and 

Missed  step 

Reinstall 

DTA 

Performed  step 
incorrectly 

1 

Looked  in  wrong 
location 

1 

Identified  part 
incorrectly 

2 

3 

Went  to  wrong 
section  of  TI 

1 

Install 

Missed  step 

1 

2 

Rudder 

Protractor 

Performed  step 
incorrectly 

2 

4 

1 

Looked  in  wrong 
location 

4 

5 

3 

Identified  part 
incorrectly 

1 

6 

Went  to  wrong 
section  of  TI 

1 

Check  and 

Missed  step 

Adjust 

Rudder  Trim 

Performed  step 
incorrectly 

4 

7 

3 

Looked  in  wrong 
location 

1 

3 

2 

Identified  part 
incorrectly 

1 

Went  to  wrong 
section  of  TI 

2 

2 

Total 

16 

31 

16 

Number  of  subjects 

3 

6 

2 

Mean  errors/subject 

5.3 

5.2 

8.0 
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Table  10.  Correetive-mainteiiaiiee  erran:  experienced  subjects. 


NTIPS(E)  NTIPS(P)  Conv(P) 


Remove  and 

Missed  step 

Reinstall 

DTA 

Performed  step 
incorrectly 

2 

Looked  in  wrong 
location 

Identified  pert 
incorrectly 

2 

Went  to  wrong 
section  of  TI 

1 

Install 

Missed  step 

Rudder 

Protractor 

Performed  step 
incorrectly 

2 

1 

Looked  in  wrong 
location 

4 

5 

Identified  part 
incorrectly 

4 

3 

4 

Went  to  wrong 
section  of  TI 

Check  and 

Missed  step 

1 

Adjust 

Rudder  Trim 

Performed  step 
incorrectly 

6 

4 

1 

Looked  in  wrong 
location 

1 

1 

Identified  part 
incorrectly 

3 

1 

Went  to  wrong 
section  of  TI 

3 

Total 

20 

14 

15 

Number  of  subjects 

5 

4 

4 

Mean  errors/subject 

4.0 

3.5 

3.75 
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Table  11.  Coereetfaermelnteneiiee 


eomhined  for  ell  subjeets. 


NT1P8(E)  NnP8(P) 


Remove  and 

Missed  step 

Reinstall 

DTA 

Performed  step 
incorrectly 

1 

2 

Looked  in  wrong 
location 

Identified  part 
incorrectly 

2 

S 

Went  to  errong 
section  of  TI 

2 

Install 

Missed  step 

1 

Rudder 

Protractor 

Performed  step 
incorrectly 

4 

S 

Looked  in  wroi^ 
location 

8 

5 

Identified  part 
incorrectly 

S 

9 

Went  to  wrong 
section  of  TI 

Check  and 

Missed  step 

1 

Adjust 

Rudder  Trim 

Performed  step 
incorrectly 

10 

11 

Looked  in  wrong 
location 

1 

4 

Identified  part 
incorrectly 

3 

Went  to  wroi^ 
section  of  TI 

2 

Total 

36 

45 

Number  of  subjeets 

8 

10 

Mean  errors/subjects 

4.5 

4.5 

/ 


CONVCP) 

1 

2 

1 

8 

4 

1 

4 
3 
2 

5 

31 

6 

5.2 
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ALL  SUBJECTS  —  TOTAL  ERRORS  USING  EACH  MEDIUM 


Figure  4.  Corrective  maintenance  errors. 


r 


o  The  largest  percentage  of  errors  for  conventional  paper  TI  is  asso¬ 
ciated  with  looking  for  a  part  in  the  wrong  location  of  the  RMTS. 

o  Most  of  the  errors  occurred  in  three  categories  —  performed  a  step 
incorrectly,  looked  for  a  part  in  the  wrong  location  in  the  RMTS,  and 
attempted  to  perform  the  work  on  the  wrong  part. 

o  Conventional  paper  TI  showed  the  highest  percentage  of  errors 
related  to  losing  one's  place  in  the  TI  and  going  to  the  wrong  section. 

Most  of  the  errors  in  step  performance  were  associated  with  checking  and  adjusting 
the  rudder  trim.  Specifically,  these  errors  occurred  while  the  subject  was  following  the 
instructions  to  compare  the  degrees  of  rudder-trim  left  and  right  and  then  moving  to  the 
proper  one  of  two  possible  steps  depending  on  the  result;  i.e.,  branching  based  on  observed 
rudder  movement.  It  appears  that  the  NTIPS  TI,  as  presented,  was  more  difficult  to 
follow  through  this  sequence  than  was  the  conventional  TI.  In  the  other  major 
performance  task,  installing  the  rudder  protractor,  the  greatest  number  of  errors  was  in 
going  to  the  wrong  location  for  attachii^  the  protractor  to  the  rudder  and  in  identifying 
parts  incorrectly.  Observations  of  subjects  performing  these  tasks  led  to  the  conclusion 
that  neither  the  graphic  nor  the  text  TI  clearly  identified  the  parts  of  the  protractor. 


4.4  PREFERENCE  DATA  AND  TI  EVALUATION 

A  preference  questionnaire  (Appendix  B)  was  completed  by  all  subjects.  The 
subjects  were  instructed  to  express  a  preference  as  to  paper  TI  or  TI  delivered 
electronically  and  to  rate  features  of  the  TI  and  the  electronic  delivery  device  on  a  scale 
from  1  to  5,  with  1  being  poor  and  S  being  excellent.  The  categories  in  the  questionnaire 
are  as  follows: 

o  Troubleshooting 

-  Fault  verification 

-  Test  instructions 

-  Graphics 

-  RMTS  Diagrams  (did  not  use  in  test) 

0  Corrective  Maintenance 

-  Introductory  discussion 

-  Set-up  instructions 

-  General  safety  instructions 

-  Step  instructions  (text  and  graphics) 

-  Support  TI 

o  Electronic  System  Features 

-  Keyboard  features 

-  Touch  screen  features 

-  Screen  display  features  (brightness,  readability,  etc.) 

-  Procedures  for  moving  through  TI. 
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Questions  presented  to  elicit  an  expression  of  preference  for  electronic  delivery  or 
for  paper  TI  delivery  were  as  follows: 

1.  If  you  had  a  choice  of  using  an  electronic  or  paper-based  manual  to  perform  tasks, 

which  would  you  choose? _ _ _ 

2.  In  working  with  the  technical  documentation,  was  it  easier  with  the  electronic 

device? _ Or  paper? _ _ 

3.  Which  mode  of  presentation  was  better  organized  for  your  purposes?  Electronic? 

_ Or  paper? _ _ _ 

Table  12  summsurizes  the  responses  to  these  questions,  broken  down  by  experience 
level  of  the  technicians.  As  may  be  seen  from  Fig.  2  (Section  3),  20  of  the  24  test 
subjects  performed  tests  with  a  form  of  electronically  delivered  TI:  They  either  used 
FIND  or  they  used  corrective  maintenance  TI  that  was  electronically  delivered. 

Table  12.  Subjects'  preference  for  TI  medium:  paper  or  electronic-delivery. 


Experienced 

Paper  Electronic 

Inexperienced 

Paper  Electronic 

Paper 

Total 

Electronic 

Question  1 

1 

10 

1 

8 

2 

18 

Question  2 

2 

9 

1 

8 

3 

17 

Question  3 

1 

10 

2 

7 

3 

17 

Of  the  entire  population  of  test  subjects,  90  percent  preferred  electronic  display  to 
paper  TI  based  on  their  recent  experience  with  it.  Of  the  total  group  85%  considered  the 
TI  they  had  used  to  have  been  better  organized  and  easier  to  use  in  electronic  form  than 
in  paper  form.  It  is  noted  that  these  evaluations  were  made  in  spite  of  the  fact  that  the 
hardware  used  to  present  the  TI  electronically  was  far  from  optimally  designed  for 
flight-line  maintenance  and  the  fact  that  the  technicians  recognized  a  number  of  specific 
defects  in  the  TI  itself  (e.g.,  graphics  quality  and  presentation  sequence  in  FIND). 

The  subjects'  responses  to  the  questionnaire  were  summarized  by  groups  based  on 
the  TI  types  used  in  the  field  test.  Table  13  shows  the  mean  ratings  for  troubleshooting  TI 
by  subjects  using  FIND  as  jmpared  to  ratings  by  subjects  using  conventional  paper  TI. 
The  following  points  can  be  made  from  this  table: 
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o  For  the  actual  presentation  of  text  and  graphics,  inexperienced 
subjects  rated  conventional  paper  TJ  somewhat  higher  than  FIND; 
they  were  least  satisfied  with  the  graphics  included  in  FIND  (an 
average  rating  of  3.1). 

o  Experienced  subjects  rated  the  text  and  graphics  presentations  of 
FIND  higher  than  that  of  conventional  paper;  they  were  least 
satisfied  with  the  text  in  the  conventional  TI  (an  average  rating  of 
2.2). 

The  text  and  graphics  in  FIND  provided  detailed  step-by-step  instructions  for 
identifying  and  removing  the  access  cover,  locating  the  connector  plug,  and  performing 
the  selected  tests.  Inexperienced  subjects  found  it  more  difficult  to  read  and  follow  the 
graphics  than  did  the  experienced  vsubjects.  Table  14  shows  the  characteristics  of  FIND 
that  were  liked  most  and  least  by  each  subject  group.  Essentially,  the  inexperienced 
group  liked  the  step-by-step  instructions  and  the  organization  of  procedure  best,  while  the 
experienced  group  found  these  characteristics  to  be  the  least  desirable.  Observations  of 
troubleshooting  performance  suggest  that  experienced  subjects  were  slowed  down  by  the 
FIND  procedure  of  sequentially  presenting  one  test  or  check  at  a  time;  this  was  not  the 
case  for  inexperienced  subjects.  Approximately  one-third  of  all  subjects  considered  that 
FIND  would  be  most  useful  for  complex  or  unfamiliar  troubleshooting  problems.  On  the 
other  hand,  the  procedural  nature  of  FIND  takes  the  technician  out  of  any  real  decision¬ 
making  role  in  the  problem-solving  process  —  it  does  not  provide  the  conceptual 
framework  for  understanding  and  interpreting  the  troubleshooting  problem  but  converts  it 
to  a  routine  sequential  procedure  for  which  it  provides  detailed  direction.  Obviously  such 
an  approach  would  appeal  more  to  inexperienced  technicians  than  to  experienced 
technicians  who  consider  that  they  are  very  familiar  with  the  system  involved. 


Table  13.  Troubleshooting  TI. 
(Mean  preference  ratings  on  a  scale 
of  1  (poor)  to  5  (excellent)) 


FIND  Conventional  Paper 

Experienced  Inexperienced  Experienced  Inexperienced 


Text  Quality  3.9  4.2  2.2  4.5 

Graphics  4.1  3.1  3.6  4.4 

Quality 
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Table  14.  FIND  characteristics  and  subject  preferences. 


Experienced 

Most  liked  -  Touch  screen 

-  Menus 


Least  liked 


-  Step  by  step 
procedures 


Inexperienced 


-  Step  by  step 
procedures 

-  Organization 
(order)  of 
procedure 

-  Graphics 
quality 
(detail) 


-  Organization 
(order)  of 
procedure 


-  System 

response  time 


Table  15  shows  the  subjects'  ratings  for  the  three  types  of  corrective-maintenance 
TI.  These  ratings  range  from  2.9  -  3.0  to  4.5,  with  the  lowest  ratings  given  to  graphics  in 
NTIPS  paper  TI  (2.9)  and  NTIPS  electronic-delivery  TI  (3.0)  and  the  highest  to  text  in 
NTIPS  electronic  (4.5).  For  the  most  part,  the  text  of  procedural  steps  was  rated  higher 
for  NTIPS  TI  than  for  conventional  paper  TI. 


Table  IS.  Corrective  maintenance  TI. 
(Mean  preference  ratings  on  a  scale 
of  1  to  5  (poor  to  excellent)) 


Instruction 

NTIPS(E) 

NTIPS(P) 

CONV(P) 

E 

I 

E 

I 

E 

I 

Set  up 

3.8 

4.1 

3.9 

3.4 

3.9 

3.3 

Steps  (Text) 

4.2 

4.5 

4.3 

3.7 

3.3 

3.9 

Graphics 

3.0 

4.1 

3.8 

2.9 

3.5 

3.8 

Table  16  shows  subjects'  selection  of  NTIPS  electronic-delivery  characteristics  that 
were  the  most  and  least  effective  in  assisting  the  corrective-maintenance  process.  All 
subjects  agreed  that  size  and  detail  of  graphics  were  the  least  attractive  features.  The 
features  selected  as  most  desirable  were  the  touch  screen,  the  size  of  the  text,  the 
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availability  of  a  screen  printer,  and  the  ability  to  move  around  in  the  data  (i.e.,  flexibility 
of  access  to  particular  pieces  of  information). 

Table  18.  NTIPS  eleetroide  deUvery  Tl  characteristics  and  sub]ect  preferences. 

Experienced  Inexperienced 


Most  Liked 


Least  Liked 


-  Touch  screen  -  Touch  screen 

-  Availability  of  screen  printer  -  Size  of  text 

-  Ability  to  move 
around  in  data 
(flexibility  of  data  areas) 

-  Size  of  graphics  -  Number  of  graphics 

-  Detail  of  graphics  -  Detail  of  graphics 


In  the  third  section  of  the  questionnaire,  subjects  rated  electronic  delivery-system 
features  such  as  the  keyboard,  the  touch  screen,  the  screen  layout,  and  the  menu 
structure.  As  shown  in  Table  17,  all  the  features  received  high  ratings  for  FIND  and  for 
NTIPS  electronically  delivered  corrective-maintenance  TI;  the  range  was  from  3.5  to  4.6. 

Table  17.  Electronic  delivery  system  features. 

(Mean  Preference  Ratings  on  a  scale 
of  1  to  5  (poor  to  excellent)) 


Features 

FIND 

NTIPS(E) 

Experienced 

Inexperienced 

Experienced 

Inexperienced 

Keyboard 

4.2 

4.1 

4.5 

4.2 

Touch  screen 

3.5 

4.2 

4.6 

4.4 

Screen  layout 

4.2 

4.1 

4.3 

4.1 

Menu  structure 

4.0 

4.2 

3.3 

4.1 

Finally,  subjects  were  asked  to  provide  their  reasons  for  preferring  electronic 
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delivery  of  TI  and  to  suggest  specific  areas  for  improvement.  The  most  positive 
comments  about  electronic  delivery  were  that  (1)  it  provides  easier  access  to  TI,  (2)  it 
saves  space,  and  (3)  electronically  stored  and  delivered  TI  is  easier  to  update.  Suggestions 
for  improvement  of  NTIPS  TI  included  (1)  increasing  the  size  and  detail  of  graphics, 
(2)  increasing  the  size  of  touch  labels,  and  (3)  including  caution  notes  on  the  same  screen 
with  the  relevant  maintenance  step.  Table  18  summarizes  these  comments  and  shows 
their  frequency  from  each  experience  level  group. 
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Table  18.  Subleets*  comments  on  acceptability  and  suggestions 
t<x  imj^oving  electronic  delivery  of  H. 


Comments 


Experienced 


Inexperienced 


Electronic-delivery  provides 
easier  access  to 
desired  sections 

8 

5 

Electronic-delivery  saves  space 

2 

1 

Electronic-delivery  TI  is  easier 
to  update 

2 

The  touch  screen  makes  the  job 
go  faster 

1 

Electronic  presentation  is  good 
for  self  teaching 

1 

Graphics  are  hard  to  read  - 
should  be  larger,  more  detailed 

3 

2 

Graphics  should  be  presented 
with  positive  image  (black  on 
white) 

1 

Caution  notes  should  appear 
with  step  (same  screen) 

1 

Touch  labels  are  too  close 
together 

1 

1 

FIND  is  time  consuming 

1 

1 

Total  number  of  comments 

20 

11 

Total  number  of  technicians  in  class 

11 

9 
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5.0  CONCLUSIONS 


5.1  INTRODUCTION 

The  NTIPS  test  and  evaluation  was  conducted  in  two  stages: 

1.  The  preparation  of  test  TI  in  accordance  with  NTIPS  specifications  and  by  use  of 
an  NTIPS-developed  automated  authoring  system. 

2.  The  comparison  of  technician  performance  quality  resulting  from  the  use  of 
NTIPS  and  conventional  TI  in  carrying  out  maintenance  tasks  on  an  F-14A  aircraft. 
Conclusions  drawn  from  the  data  collected  during  both  stages  are  presented  in  the 
following  sections. 

5.2  SUMMARY  OF  CONCLUSIONS 

Results  of  the  NTIPS  field  test  have  shown  that  TI  constructed  by  automated 
authoring  and  according  to  NTIPS  specifications,  when  applied  by  fleet  technicians  in 
operational  maintenance  tasks,  can  significantly  improve  performance  (particularly 
troubleshooting).  These  results  also  show  that  the  electronic  presentation  of  maintenance 
TI  is  considered  superior  to  paper  presentation  of  TI  by  approximately  90%  of  experienced 
and  inexperienced  technicians. 

The  tests  also  provided  valuable  guidance,  both  in  the  area  of  specifying  the  most 
effective  TI  (e.g.,  the  need  for  better  graphics,  better  organization,  better  approaches  to 
presentation)  and  in  the  area  of  electronic-presentation  approaches.  A  number  of  these 
suggestions  have  already  been  incorporated  into  NTIPS  approaches. 

Although  it  is  difficult  or  impossible  to  generalize  from  a  test  involving  a  small 
population  of  technicians  working  on  a  specific  maintenance  area  (an  aircraft  electro¬ 
mechanical  system),  it  appears  there  is  no  fleet  objection  to  the  automated  generation 
and  presentation  of  fault-isolation  and  corrective-maintenance  TI.  The  TI  innovations 
proposed  by  NTIPS  in  TI  content,  format,  style,  and  organization  are  as  effective  and  in 
some  cases  more  effective  than  conventional  paper  TI. 

5.3  CONCLUSIONS  REGARDING  NTIPS  SPECIFICATIONS  AND 

AUTHORING  SYSTEM  FOR  GENERATING  TI 

In  general,  the  TI  contractor  was  able  to  follow  the  NTIPS  TI  specifications  and  was 
also  able  to  use  the  TI  authoring  system  to  prepare  the  experimental  TI  presenting  F-14A 
procedures  used  in  the  field  test.  Use  of  an  automated  authoring  system  to  prepare  Navy 
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weapon  system  TI  in  no  way  reduces  the  need  for  a  careful  contractually  mandated  quality 
assurance  program.  Specific  obeervations  recorded  during  the  preparation  process  are 
summarized  below. 

5.3.1  ^peeifieatkms 

No  problems  were  found  regarding  clarity  of  specifications.  However,  test  date 
showed  that  the  contractor  experienced  consistent  problems  in  carrying  out  some  of  the 
aspects  of  TI  generation  required  by  TI  specifications.  Since  some  of  the  NTIPS 
approaches  to  TI  generation  were  entirely  new  to  the  TM  writers,  this  was  not  an 
unexpected  result. 

5.3.2  Computer-Assisted  Authoring 

During  the  TI  generation  process,  extensive  interaction  was  required  between  the  TI 
contractor  and  the  developer  of  the  authoring  programs  due  to  the  radical  difference 
between  the  automated  approach  and  the  manual  approach  to  TI  generation  with  which 
the  contractor  was  familiar.  The  prompting  feature  of  the  NTIPS  automated  authoring 

systems  was  not  particularly  effective  as  designed.  The  computer  used  to  host  the 

% 

authoring  routine  has  been  made  obsolete  by  rapid  technological  progress  in  automated 
authoring. 

5.4  CONCLUSIONS  AND  OBSERVAHONS  BASED  ON  DATA  FROM  FIELD  TEST 

This  section  summarizes  the  impact  of  the  NTIPS  experimental  TI  on  maintenance 
task  performance.  This  section  also  summarizes  observations  made  during  the  operational 
field  test  conducted  at  the  Miramar  Naval  Air  Station  including 

(1)  Observations  and  specific  recommendations  made  by  the  test  subjects  con¬ 
cerning  their  reactions  to,  and  proposals  for  improvement  of,  the  NTIPS 
experimental  TI  and  the  electronic-delivery  methodology. 

(2)  Observations  and  recommendations  made  by  test  coordination  personnel  con¬ 
cerning  the  field  use  of  NTIPS  and  conventional  TI,  and  proposed  methods  of 
improving  NTIPS  TI  and  delivery  methodology. 

Reactions  of  test  subjects  to  the  various  TI  features  were  consistent  with  opera¬ 
tional  tests  made  previously  by  the  three  services  or  various  kinds  of  TI  content,  format, 
style,  medium,  and  procedural  organization.  Special  relevance  of  these  reactions  to 
improvement  of  the  NTIPS  technology  (both  TI  and  presentation  methodology)  is  cited  in 
the  following  section. 
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5.4.1  Different  Apfiroacbes  to  TI  Use 

Several  approaches  to  the  use  of  Tl  were  observed  among  technicians  during  the 
test.  A  popular  approach  involved  a  complete  review  of  the  procedure  before  any  steps 
were  performed,  instead  of  reading  each  step  once  and  then  performing  it  before  moving 
aliead  in  the  TI.  The  implication  of  this  observation  is  the  possible  benefit  which  would 
result  from  providing  a  browsing  mode  in  the  electronic  delivery  of  task  instructions,  i.e., 
providing  a  summary  of  a  sequence  of  steps  (for  example,  a  kind  of  annotated  checklist)  as 
an  alternative  to  complete  details  for  one  step  at  a  time. 

5.4.2  Troubleshooting  Task  Performance  Time 

For  experienced  subjects,  use  of  FIND  increased  total  fault  isolation  time  an 
average  of  37%,  caused  entirely  by  the  necessity  to  peruse  each  sequence  presented  by 
the  FIND  screen  rather  than  skimming  over  a  schematic  with  a  speed  made  possible  by 
experience.  Thus,  experienced  technicians  spent  2.4  times  longer  perusing  the  FIND  TI 
than  the  conventional  schematic  fault-isolation  TI.  The  actual  working  time  (testing, 
locating,  and  exposing  test  points,  etc.)  was  reduced  by  an  average  of  28.2%  by  use  of 
FIND. 

For  inexperienced  technicians,  the  use  of  FIND  decreased  slightly  (9.2%)  the  total 
fault-isolation  time  over  that  resulting  from  use  of  conventional  TI.  (Troubleshooting 
with  conventional  TI  by  inexperienced  technicians  was,  as  noted  above,  entirely 
unsuccessful.)  With  FIND,  inexperienced  technicians  completed  their  tasks  in  about  the 
same  time  as  did  experienced  technicians.  Inexperienced  technicians  took  7.3%  longer. 
As  in  the  case  of  the  experienced  technicians,  actual  work-performance  time  was  reduced 
(by  36.2%)  through  use  of  FIND. 

5.4.3  Errors  in  Troubleshooting 

The  most  significant  error  in  fault  isolation  was  a  failure  to  isolate  the  introduced 
fault.  None  of  the  technicians,  experienced  or  inexperienced,  failed  to  locate  the  fault 
using  FIND.  All  inexperienced  and  2  of  the  7  experienced  technicians  failed  to  isolate  the 
fault  with  conventional  fault-isolation  (schematic)  TI.  Occurrence  of  other  types  of 
errors  was  about  the  same  for  troubleshooting  by  experienced  technicans  using  FIND  and 
using  conventional  TI;  inexperienced  technicians  committed  about  twice  as  many  errors 
(10/5)  using  conventional  TI  as  using  FIND. 
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5.4.4  CkureetiverMafateiianee  Perfonnanee  Time 

As  in  the  ease  of  the  troubleshooting  tasks,  corrective-maintenance  tasks  performed 
with  NUPS  TI  (both  paper  and  electronieall^rprcsented)  took  experienced  technicians 
somewhat  longer  than  performance  with  conventional  Tl  (17.6%  and  12.6%,  respectively). 
This  time  increase  is  ascribed  to  the  pow  quality  of  graphics  and,  for  the  subjects  using 
the  electronic  device,  to  the  need  to  move  back  and  forth  between  the  device  and  their 
work  location.  Time  spent  actually  perusing  the  TI  was  again  greater  for  both  types  of 
NTIPS  TI  than  for  conventional  TI  (7.4%  and  27.8%  for  NTIPS  paper  and  electronically- 
presented  TI,  respectively).  The  actual  worktime  was  approximately  the  same  (rangii^ 
from  31.0  minutes  to  36.9  minutes).  On  the  average,  inexperienced  technicians  took 
longer  to  perform  their  corrective-maintenance  tasks  using  NTIPS  electronic  TI  (and 
committed  fewer  errors)  than  with  conventional  TI.  The  relative  average  total  times 
involved  were  77.66  minutes  vs.  62.48  minutes,  but  a  significant  fraction  of  the  NTIPS 
electronic  result  was  due  to  abnormally  long  work-time  of  a  single  technician.  If  his 
contribution  to  the  total  time  for  NTIPS  electronic  TI  use  was  removed,  the  total 
performance  time  for  the  three  types  of  TI  used  would  be  much  closer  (68.0  minutes  for 
NTIPS  electronics,  59.15  minutes  NTIPS  paper,  62.5  minutes  conventional).  Breakdowns 
of  corrective-maintenance  time  amot^  the  three  components  used  (Tl-perusal,  hands-on 
work,  and  miscellaneous)  did  not  show  significant  differences  except  in  the  case  of  the 
technician  referred  to  above. 

5.4.5  Cwrective-Maintenance  Errors 

On  the  average,  inexperienced  technicians  committed  fewer  errors  per  technician 
with  both  types  of  NTIPS  TI  than  with  conventional  TI  in  corrective-maintenance  tasks 
(about  35%  less).  There  was  little  difference  in  the  error  rates  for  experienced 
technicians  among  the  three  types  of  Tl;  these  rates,  as  might  be  expected,  were  lower 
for  experienced  technicians  than  for  inexperienced  technicians. 

5.4.6  Range  in  Reading  Abilities 

Test  subjects  represented  a  wide  range  of  reading  abilities.  Some  subjects  read 
aloud,  others  by  following  along  with  their  fingers.  Reading  rates  and  apparent  compre¬ 
hension  varied  from  fast  with  good  comprehension,  to  slow  with  many  re-reads.  English 
was  the  second  language  for  some  of  the  subjects.  Implications  to  TI  design  of  the 
readii^trait  variations  observed  during  the  tests  are  under  further  study. 


I 


/ 


52 


5.4.7  Preferences  for  TI  Types 

During  the  troubleshooting  tests,  those  technicians  assigned  to  the  conventional  TI 
category  were  asked  to  proceed  with  the  task  using  the  method  they  normally  used  in 
carrying  out  squadron  maintenance.  Most  skipped  or  abbreviated  the  fault  verification 
task  and  many  took  notes  on  the  relevant  portions  of  the  Level  2C  troubleshooting 
schematics  of  NAVAIR  01-F-14AAA-2-2-16.3.  These  notes  were  taken  to  the  job  rather 
than  the  full  (8-1/2”  x  34”)  schematics.  During  debriefing,  a  pattern  of  preference  as  a 
function  of  experience-level  became  apparent  for  the  technicians  in  this  group.  All 
technicians  using  conventional  TI  were  familiarized  also  with  the  NTIPS  FIND  method  of 
troubleshooting  prior  to  the  test.  In  spite  of  the  high  (90%)  preference  for  electronic 
delivery  expressed  by  test  subjects  in  answer  to  direct  questions,  a  number  of  experienced 
technicians  expressed  a  preference  for  the  familiar  troubleshooting  schematics  as  a 
troubleshooting  approach,  while  the  inexperienced  technicians  favored  the  step-by-step 
procedures  of  FIND.  Previous  research  on  this  question  has  suggested  that  a  proce- 
duralized  format  such  as  FIND  (do  this,  then  do  that)  and  a  decision-making  format  such 
as  the  schematics  included  in  the  conventional  TI  might  be  integrated,  particularly  with 
electronic  display.  Information  might  also  be  added  to  help  the  inexperienced  technicians 
transition  from  the  proceduralized  to  the  decision-making  format.  Further  research  is 
needed  to  determine  the  best  way  to  handle  schematics  on  an  electronic  display  device. 

Both  experienced  and  inexperienced  technicians  favored  the  electronic  delivery  of 
corrective-maintenance  TI.  They  considered  that  the  electronic  delivery  was  superior  to 
paper  because  it  provides  easier  access  to  desired  TI  sections,  saves  storage  space,  and  is 
easier  to  update. 

5.4.8  Communication  and  Team  Performance 

The  NTIPS  TI,  for  both  corrective  and  troubleshooting  tasks,  provided  only  minor 
guidance  for  work  allocation  among  the  various  members  of  a  maintenance  team;  the 
setup  instructions  provided  did  not  detail  requirements  for  communication  between  and 
among  workers.  Safety  regulations  required  that  cranial  protection,  which  constrained 
normal  conversation,  be  worn  during  aircraft  maintenance.  Future  TI  specifications 
should  emphasize  (1)  the  identification  of  worker  responsibility  in  a  task-specific  manner 
and  of  worker  locations,  and  (2)  the  need  for,  and  the  means  of,  communication  between 
workers. 
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5.4.9  Gbraphies  Detail 

Many  of  the  graphics  in  the  NTIPS  TI  (especially  the  electronic  version)  showed 
necessary  detail*  but  with  dimensions  that  were  too  small  to  be  of  greatest  benefit  to  the 
user.  The  quality  of  the  graphics  was  below  the  quality  obtainable  by  modem  state  of  the 
art  systems.  The  result  was  unnecMsarily  high  error  rates  (looking  in  the  wrong  location 
for  a  part  and  identifying  the  wrong  part).  High  quality  gri^hics  are  required,  both  in 
terms  of  size  and  fidelity  of  detail,  for  future  implications.  Ultimately,  optimization  of 
text-graphics  modules  involves  trade-offs  among  such  variables  as  (1)  the  relationships  of 
text  to  graphics  on  a  given  display,  (2)  the  determination  of  the  ideal  amount  of  work 
prescribed  by  the  instructions  in  a  single  frame,  (3)  the  resolution  of  the  display,  (4)  the 
f  ield-of-view  of  the  graphic,  and  (S)  the  level  of  detail  provided  in  the  graphic.  Rules-of- 
thumb  and  conventions  based  on  human  factors  engineering  are  available  for  the 
treatment  of  all  these  variables.  Appropriate  guidance  must  be  incorporated  in  a  clearly 
interpretable  form  in  future  specifications  for  automated  generation  of  TI. 

5.4.10  Decision-Making  Performances 

One  of  the  test  tasks  (Check  and  Adjust  Rudder  Trim)  required  branching;  e.g., 
statements  of  the  following  type:  "If  greater  than  7  degrees,  go  to  step  x;  if  less  than  7 
degrees,  go  to  step  y."  Observations  by  data  recorders,  analysis  of  time  spent  by 
technicians  in  perusing  the  TI,  and  analysis  of  errors  all  indicated  that  technicians  had 
trouble  understanding  these  sections  of  the  TI  no  matter  what  the  form  of  presentation. 
The  cause  and  solution  of  this  problem  are  not  clear.  Further  analysis  leading  to 
improvement  of  the  relevant  specifications  is  necessary  to  solve  this  important  problem, 
especially  for  the  electronic  delivery  of  TI. 

5.4.11  Electronic-Delivery  Device 

The  device  used  in  the  field  test  was  an  off-the-shelf  (non-militarized)  item  and 
since  its  significance  was  solely  to  provide  a  basis  for  evaluation  of  electronically 
presented  TI,  reactions  to  the  display  device  itself  received  from  test  subjects  are  not 
intrinsically  significant.  However,  some  observations  were  general  enough  to  be  useful. 
For  example,  the  use  of  powered  devices  on  the  aircraft  during  maintenance  is  prohibited. 
If  a  portable  device  is  to  be  taken  aboard  an  aircraft  as  a  maintenance  tool,  this  question 
must  be  dealt  with.  Glare,  even  in  a  hangar  environment,  presents  a  problem;  a  delivery 
device  must  be  carefully  constructed  fot  optimum  TI  visibility.  Some  means  of  anchoring 
the  device  is  required  (few  flat  surfaces  are  available).  Additionally,  although  technicians 


commented  favorably  about  the  concept  of  the  touch,  some  drawbacks  were  noted,  e.g., 
the  touch  labels  were  too  small  and  too  close  together. 


5.4.12  Paper-Delivery  Medium 

Deficiencies  and  problems  noted  for  the  electronic-display  medium  in  general  apply 
also  to  paper.  In  addition,  observations  unique  to  paper  include  (1)  the  critical  need  for  a 
comprehensible  and  consistent  TI  numbering  system,  (2)  a  means  of  binding  that  allows  the 
package  to  lie  flat  during  use,  and  (3)  a  means  of  packaging  sets  of  task  procedures  that 
retains  the  benefits  of  smallness  and  portability. 
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AfPBHDIZ  Ai 

DATA  COLLBCnON  POEMS 


Form  A:  Activity  Time  **  Corrective  Meintenence  57 
Form  B:  Errors  *  Corrective  Meintenence  58 

Filled-Out  Form  A  A  B  for  Tesk  4~2.5,  Subtesk  1  59 

A  semple  dets^coUeetion  form  for  e  eorreet!ve*meintenenee  tesk  is  included  ss  page 
56  (Activity  Time)  end  page  57  (Error).  Forms  of  this  type  were  filled  out  by  test 
personnel  for  each  subtask  of  each  maintenance  task,  while  the  test  subject  was 
performing  the  task. 

Filled-out  Activity-Time  end  Errors  forms  for  Task  4-2.5,  Install  Rudder  Protractor 
Subtask  1:  Zero  Rudder  Protractor  (using  KTIPS  paper)  are  also  incorporated  as  pp.  58 
and  59.  The  symbols  used  to  describe  the  individual  actions  being  carried  out  during  task 
performance  are  described  in  Section  3.6. 


APPENDIX  A 

DATA  COLLECTION  FORMS 


Form  A:  Activity  Time  -  Corrective  Maintenance  58 

Form  B:  Errors  -  Corrective  Maintenance  59 

Filled-Out  Form  A  &  B  for  Task  4-2.5,  Subtask  1  60 

A  sample  data-collection  form  for  a  corrective-maintenance  task  is  included  as  page 
57  (Activity  Time)  and  page  58  (Error).  Forms  of  this  type  were  filled  out  by  test 
personnel  for  each  subtask  of  each  maintenance  task,  while  the  test  subject  was 
performing  the  task. 

Filled-out  Activity-Time  and  Errors  forms  for  Task  4-2.5,  Install  Rudder  Protractor 
Subtask  1:  Zero  Rudder  Protractor  (using  NTIPS  paper)  are  also  incorporated  as  pp.  59 
and  60.  The  symbols  used  to  describe  the  individual  actions  being  carried  out  during  task 
performance  are  described  in  Section  3.6. 
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f-ORM  A;  ACTIVITY  I  IME  -  CORRECTIVE  MAINTENANCE 


FORM  A:  ACTIVITY  TIME  -  CORRECTIVE  MAINTENANCE 


PREFERENCE  QUESTIONNAIEE 
Instructions  for  Completing  Questionnaire 


Now  that  you  have  done  troubleshooting  and  corrective  maintenance  with  different 
forms  of  technical  information  (TI),  we  are  interested  in  your  evaluation.  First,  read  the 
information  on  this  page.  Then  complete  the  questionnaires  that  follow.  Be  sure  to 
complete  every  item.  Don't  leave  any  items  blank. 

Your  questionnaire  responses  will  not  be  used  to  rate  your  fitness.  All  of  your 
responses  will  be  kept  in  total  confidence. 

Some  questionnaire  items  ask  you  to  rate  TI  features  on  a  five-point  scale  from  1 
(for  Very  Poor)  to  S  (for  Excellent).  Use  the  definitions  of  the  numbers  in  the  scales  that 
are  given  below  to  help  make  your  rating  decisions. 

1  -  VERY  POOR  I  don't  see  how  the  job  can  be  done  with  this  feature  the 

way  it  is. 

2  -  POOR  This  feature  isn’t  very  good. 

3  -  AVERAGE  This  feature  is  O.K. 

4  -  GOOD  This  feature  makes  tasks  easier/quicker  to  perform. 

5  -  EXCELLENT  This  feature  is  really  great. 

As  you  come  to  the  lists  of  features  on  the  questionnaires,  try  to  remember  how 
much  each  feature  helped  or  hindered  you.  Select  the  rating  that  corresponds  to  your 
judgment,  and  mark  it  on  the  questionnaire. 

Use  the  COMMENTS  column  at  the  right  of  rating  scales  or  at  the  end  of  it  to  note 
any  strong  feelings  about  a  feature  or  to  suggest  how  it  might  be  improved. 

Be  sure  to  complete  the  BIOGRAPHICAL  section  at  the  end  of  the  questionnaire. 

If  you  have  any  questions,  ask  a  data  collector  for  help. 

POINTS  TO  CONSIDER 

Please  follow  the  rules  listed  below  when  completing  the  questionnaire. 

1.  As  you  come  to  characteristics  on  the  questionnaire,  try  to  remember 
how  much  each  characteristic  helped  or  hindered  you  during  your  task 
performance.  Select  the  rating  that  corresponds  to  this  degree  of 
helpfulness  and  mark  it  on  the  questionnaire. 

2.  If  you  have  strong  feelings  about  some  characteristic  of  the  information 
you  worked  with,  but  this  characteristic  is  not  listed  on  the 
questionnaire,  please  describe  and  rate  it  (or  them)  in  the  comments 
section. 

3.  Please  complete  the  administrative  section  (last  page)  of  the 
questionnaire.  If  you  have  any  questions,  ask  the  data  collector. 
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APPENDIX  B 

PREFERENCE  QUESTIONNAIRE 


Appendix  B  contains  a  sample  preference  questionnaire  filled  out  following  the  test 
by  all  of  the  24  test  subjects.  Results  are  analyzed  and  discussed  in  Section  4.4.  The 
complete  set  of  executed  questionnaires  has  been  retained  by  the  NTIPS  Office.  Further 
analyses  of  these  data  and  those  from  the  videotaped  post-test  interviews  are  being 
carried  out  to  make  certain  that  all  possible  lessons  learned  from  this  field  test  are 
applied  to  future  NTIPS  designs. 

The  Preference  Questionnaire  consists  of  the  following  sections 


1.  Instructions  for  Completing  Questionnaire  64 

2.  Part  I  -  TECHNICAL  INFORMATION  65 

CORRECTIVE-MAINTENANCE  TI  65 

TROUBLESHOOTING  TI  67 

SUPPORT  TI  68 

3.  Part  II  ELECTRONIC  SYSTEM  FEATURES  69 

BEST  AND  WORST  FEATURES  OF  ELECTRONIC 
DELIVERY  OF  TECHNICAL  INFORMATION  71 

4.  Part  III.  Biographical  Information  72 


5.  Interview 
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PBKFERENCB  QTJB8TIONNAIRB 
Instructions  for  Completing  Questionnaire 


Now  that  you  have  done  troubleshooting  and  corrective  maintenance  with  different 
forms  of  technical  information  (Ti),  we  are  interested  In  your  evaluation.  First,  read  the 
information  on  this  page.  Then  complete  the  questionnaires  that  follow.  Be  sure  to 
complete  every  item.  Don't  leave  any  items  blank. 

Your  questionnaire  responses  will  not  be  used  to  rate  yotir  fitness.  All  of  your 
responses  will  be  kept  in  total  confidence. 

Some  questionnaire  items  ask  you  to  rate  TI  features  on  a  five-point  scale  from  1 
(for  Very  Poor)  to  S  (for  Excellent).  Use  the  definitions  of  the  numbers  in  the  scales  that 
are  given  below  to  help  make  your  rating  decisions. 

1  -  VERY  POOR  I  don't  see  how  the  job  can  be  done  with  this  feature  the 

way  it  is. 

2  -  POOR  This  feature  isn't  very  good. 

3  -  AVERAGE  This  feature  is  O.R. 

4  -  GOOD  This  feature  makes  tasks  easier/quicker  to  perform. 

5  -  EXCELLENT  This  feature  is  really  great. 

As  you  come  to  the  lists  of  features  on  the  questionnaires,  try  to  remember  how 
much  each  feature  helped  or  hindered  you.  Select  the  ratii^  that  corresponds  to  your 
judgment,  and  mark  it  on  the  questionnaire. 

Use  the  COMMENTS  column  at  the  right  of  rating  scales  or  at  the  end  of  it  to  note 
any  strong  feelings  about  a  feature  or  to  suggest  how  it  might  be  improved. 

Be  sure  to  complete  the  BIOGRAPHICAL  section  at  the  end  of  the  questionnaire. 

If  you  have  any  questions,  ask  a  data  collector  for  help. 

POINTS  TO  CONSIDER 

Please  follow  the  rules  listed  below  when  completing  the  questionnaire. 

1.  As  you  come  to  characteristics  on  the  questionnaire,  try  to  remember 
how  much  each  characteristic  helped  or  hindered  you  during  your  task 
performance.  Select  the  rating  that  corresponds  to  this  degn^ee  of 
helpfulness  and  mark  it  on  the  questionnaire. 

2.  If  you  have  strong  feelings  about  some  characteristic  of  the  information 
you  worked  with,  but  this  characteristic  is  not  listed  on  the 
questionnaire,  please  describe  and  rate  it  (or  them)  in  the  comments 
section. 

3.  Please  complete  the  administrative  section  (last  page)  of  the 
questionnaire.  If  you  have  any  questions,  ask  the  data  collector. 
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Part  I  -  Teehnieal  Infomiaticm 


Information  Characteristics  Strength  of  Approval  or  Disapproval 

1  2  3  4  5 

Very 

Poor  Poor  Avg  Good  Exc  Comments 


CORRECTIVE  MAINTENANCE  TI 

1.  Introductory  Discussion 

2.  Setup  Instructions 

a.  "Applicable  Configuration" 

b.  Test  Equipment 

c.  Tools 

d.  Materials/Parts  List 

e.  Task  References 

f.  Personnel  Required 

g.  Special  Skills  and  Knowledges 

h.  Approximate  Time  Required 

i.  List  of  Directives 

3.  General  Safety  Instructions 

4.  Step  Instructions 

a.  Organization  into  Tasks, 
Subtasks,  and  Steps 

b.  Amount  of  text 

c.  Usability  of  text 

-  Level  of  Detail 

-  Format 

-  Clarity  of  Writing 
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Part  I  (Continued) 


Information  Characteristics 


Strength  of  Approval  or  Disapproval 

1  2  3  4  5 

Very 

Poor  Poor  Avg  Good  Exc 


Comments 


CORRECTIVE  MAINTENANCE  TI 


d.  Amount  of  Graphics 


e.  Usability  of  Graphics 
-  LegibUity 


-  Understandability 


-  Size 


-  Ease  in  Finding 
Components 


-  Level  of  Detail 

-  Format 
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Part  I  (Continued) 


Information  Characteristics  Strength  of  Approval  or  Disapproval 

1  2  3  4  5 

Very 

Poor  Poor  Avg  Good  Exc  Comments 


TROUBLESHOOTING  TI 

1.  Adequacy  of  Fault  Verification 

2.  Test  Instructions 

a.  Level  of  Detail 

b.  Sequence  of  Tests 

c.  Ease  of  Access 

d.  Validity  of  Test  Actions 

3.  Usability  of  Graphics 

a.  Legibility 

b.  Understandability 

c.  Size 

d.  Help  in  locating  test  points 

e.  Level  of  detail 


f.  Format 


Part  I  (Ccmtinued) 


Information  Cbaracteriaties  Strength  of  Approval  or  Disapproval 

1  2  3  4  S 

Very 

Poor  Poor  Avg  Good  Exe  Comments 


SUPPORT  TI 

1.  Table  of  Contents 

2.  IPB 

3.  Preparatory  Instructions 

a.  Indicator  Preparation 

b.  Task  Preparation 

4.  Others  (describe) 
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Part  n  —  Eleetronie  System  Features 


Electronic  System  Features  Scale  Values 

1  2  3  4  5 

Very 

Poor  Poor  Avg  Good  Exc  Comments 


Features  of  NTIPS  Keys  and 
Keyboard 

1.  Spacing  of  keys 

2.  Arrangement  of  keys 

3.  Ease  of  operating  keys 

4.  Indication(s)  that  keys  have  been 
activated 

5.  Reliability  of  keys  (i,e.,  how  well 
did  the  keys  respond  to  use) 

Features  of  Touch-Sensitive  Screen 
Operation 

1.  Arrangement  of  touch  labels 

2.  Location  of  touch  labels 

3.  Readability  of  touch  labels 

4.  Responsiveness  of  system  to  using 
touch  labels 

General  Screen  Features 

1.  Adequacy  of  screen  size  for 
display  of  information 

2.  Brightness  of  display 

3.  Readability  of  display  screens 

4.  Contrast  between  displayed 
information  and  background 

5.  Glare  resistance  of  display  screen 
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P«rt  n  (Cmitinued) 


Electronic  System  Features  Scale  Values 

1  2  3  4  S 

Very 

Poor  Poor  Avg  Good  Exc  Comments 


Techniques  for  Controlling 
Information  Delivery 

1.  Ease  of  using  menus  to  obtain 
maintenance  information 


2.  Ease  of  returning  to  the 
appropriate  section  in  a  set  of 
procedures  after  branching  to 
obtain  additional  information 

3.  Adequacy  of  features  to  exit  from 
an  inappropriate  section  of  the 
data  base  (e.gy  following  an 
incorrect  key  press  or  equipment 
malfunction). 

4.  Adequacy  of  "prompts"  on  the 
display  for  assisting/guiding  the 
operator 

General  Comments  (use  space  below) 
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Part  n  (Continued) 

BEST  AND  WORST  CHARACTERGSTICS  OF  ELECTRONIC  DELIVERY 
OF  TECHNICAL  INFORMATION 

Please  check  the  three  Best  and  three  Worst  characteristics  of  the  electronic 
information  delivery  system  you  used  in  this  test. 

Characteristic  Best  Worst 

1 .  Step  by  step  text 

2 .  Organization  of  procedures 

3 .  Relation  of  text  to  graphics 

4 .  Size  of  text  characters 

5.  Font  (letter  style) 

6 .  Spacing  and  layout 

7 .  Size  of  graphics 

8.  Number  of  graphics 

9.  Nearness  of  graphic  to  related  text 

10.  Detail  of  graphic 

1 1 .  Graphic  callouts 

!  2.  Touch  screen 

13.  Size  of  touch  boxes 

14.  Dependability  of  touch 

15.  Size  of  the  electronic  display 

!  6.  Electronic  display  legibility 

IT.  Electronic  display  brightness 

18.  Electronic  display  glare 

19.  Printer 

20.  Menus 

21.  System  response  time 

.12.  Ability  to  move  around  in  the  data  base 


Thanks  a  lot.  If  we  missed  any  characteristics  that  you  believe  to  be  important,  pro 
or  con,  jot  them  down  below. 


Part  HI  —  Biographieal  Inf orniatloii 


Name:  _ 

Date: 

(last) 

(first) 

(middle) 

Rating: 

PavKrade: 

Civilian  Education,  Highest  Grade: _ 

Service  Schools  Completed 

(1)  _ 

(2)  _ 

(3)  _ 

Years  in  Navy:  _  Years _  Months 

Maintenance  Experience: 

Radar _  Years _  Months 

What  portion  "0"  level?: _  Years  _  Months 

What  portion  "I"  level?: _  Years  _  Months 

Any  other  radar  experience?  _  Yes  _  No 

If  yes,  please  specify:  _ 


Other  Electronic  Equipment _  Years  _  Months 

Other  Electronic  Equipment _  Years  _  Months 


Computer  Experience: 

(1)  Have  you  ever  taken  a  course  in  the  use  of  a  computer?  _  Yes _  No 

(2)  Do  you  own  a  personal  computer? _  Yes _  No 

(3)  Have  you  ever  routinely  used  a  computer  as  part  of  any  job  you  have  ever  held? 

_  Yes _  No 

*This  information  will  be  held  In  confidence  and  will  only  be  used  for  the  purpose  of  this 
study  of  TI. 


PART  m  —  GENERAL  COMMENTS  AND  PREFERENCES: 
ELECTRONIC  DEUVBRY  VS.  PAPER  DELIVERY 


1.  If  you  had  a  choice  of  using  an  electronic  or  paper-based  manual  to  perform  tasks 

which  would  you  choose? _ 

2.  In  working  with  the  technical  documentation,  was  it  easier  with  the  electronic 

device?  _ Or  paper? _ 

3.  Which  mode  of  presentation  was  better  organized  for  your  purposes?  Electronic? 

_ Or  paper? _ 

4.  What  do  you  see  as  the  major  advantages  of  the  electronic  presentation? 


INTERVIEW 


You  have  just  done  troubleshooting  using  a  conventional  TI  and  the  NTIPS  electronic 
delivery  system  for  corrective  maintenance. 


1.  Based  on  this  experience,  which  do  you  prefer?  Conventional  TI  or  NTIPS  electronic 
delivery?  Why? 

2.  How  was  your  performance  affected  using  NTIPS  electronic  delivery? 

a.  Were  any  tasks  made  easier? 

b.  Was  your  performance  more  accurate? 


c.  Did  you  work  more  rapidly? 


3,  What  problems,  if  any,  did  you  have  in  using  NTIPS  electronic  delivery? 


4.  What  problems  did  you  have  in  using  conventional  TI? 


5.  Do  you  think  that  both  troubleshooting  and  corrective  maintenance  would  be  equally 
helped,  or  hindered,  by  using  NTIPS  electronic  delivery?  Why? 


APPENDIX  C 

EXAMPLE  OF  TI  TYPES  USED  IN  TEST 


Appendix  C  includes  samples  of  each  of  the  five  types  of  Technical  Information  used 
during  the  tests.  Two  of  these  five  approaches  involved  electronic  delivery.  For 
illustration  purposes  only,  prints  from  the  display  screen  were  made  and  are  incorporated 
herewith.  These  prints  are  intended  only  to  show  the  presentation  format,  not  the  actual 
appearance  of  the  data  display. 


I.  Sample  of  Conventional  Technical  Manual  paper  troubleshooting  manual.  76 
The  schematic  displayed  is  Sheet  35,  Figure  47,  Rudder,  Stabilizer,  emd 
Spoiler  Control  S3^tem  Closed-Loop  Functional  Diagram  (Aircraft  Serial 
Number  1S982S  through  159858  and  Aircraft  Modified  by  AFC  400), 
NAVAIR  01-F14AAA-2-2-16.3,  Technical  Manual,  Organizational  Main¬ 
tenance,  Integrated  Functional  Diagrams,  Navy  Model  F-14A  Aircraft, 
Change  3,  15  June  1985.  This  schematic  was  the  TI  of  choice  for 
maintenance  technicians  performing  fault  isolation  based  on  the  fault 
symptom  produced  by  the  introduced  test  fault. 

II.  Sample  of  FIND  troubleshooting  test  sequence  (copy  prepared  by  screen  77 
prints)  pp.  C-3  -  C-5.  If  the  "good"  result  (115  Vac  from  pin  3  to  ground) 
is  obtained  from  this  test,  the  result  is  entered  into  the  computer  via 
touch  screen  and  the  program  directs  the  technician  to  the  next  logical 
test.  The  same  procedure,  of  course,  is  carried  out  for  a  "bad"  result 
(inadequate  voltage  at  pin  3) 

III.  Sample  of  NTIPS  paper  TI  showing  end  of  Task  4-2.5  (Rudder  Protractor  80 

lustration)  and  beg^inning  of  Task  4-2.6  (Check  and  Adjust  Rudder 
Positioning).  This  sample  illustrates  the  text/graphics  module  approach 
of  NTIPS  TI. 

IV.  Sample  of  NTIPS  TI  for  a  corrective  maintenance  task  (screen-printer  81 
reproduction  of  a  screen  showing  text/graphics  TI  for  a  single  step) 

V.  Sample  of  conventional  paper  work-package  TM  for  Rudder  Protractor  82 

Installation  and  Removal  (4-page  sequence)  (C-8  -  C-11).  NAVAIR  01- 
F14AAA-2-4-4.1,  Work  Package  008  03,  pp.  1-4,  15  Sept.  1981,  Organiza¬ 
tional  Maintenance. 
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Figure  47.  Rudder,  Stabilizer,  and  Spoiler  Control  System  Closed-L 


58  and  Aircraft  Modified  by  AFC  400)  (Sheet  35) 


Functional  Diag 


NAVAIR  01-F14AAA-2-2-16.3 


SAMPLE  CONVENTIONAL  TROUBLESHOOTING  PAPER  TECHNICAL  MANUAL 


n.  3-PAGE  SAMPLE  OF  A  TEST  PROCEDURE  IN  THE  FIND 
TROUBLESHOOTING  TI  (REPRODUCED  BY  SCREEN 


lUDOn  PBOmCTOS  iisuoutioi 


3.  Attach  tha  xaroad  niddar  protractor  (i)  to  tha 
varticai  tail  (in  by  tigbtaning  tha  (our  attaching 
acrawt  (2) 

4.  Chack  whathar  tha  (orward  phanoiic  apacara  (3) 
ara  (ully  againat  tha  vortical  tail  (in.  H  not.  adiuai 
tha  knuriad  aerawa  (4)  on  tha  protractor  until  tha 
gap  batwraan  tha  apacara  and  tha  tail  (in  ia  aqual 
on  both  aidaa 

END  OF  TASK 


4-2.6.  TASK  6.  CHECK  AND  ADJUST 
RUDDER  POSITIONING 

4-2.4.  ST1.  CHICK  NUODill  ZKIIO  FOSITION 


1.  Chock  that  Mtial  aquipinant  condition  liaa  boon 
aatabNahad  in  aecordanca  with  tha  (allowing  pro- 
eaduraa: 

4-2. 1 .  TASK  1 .  Ramova  Hydraulic  and  Elactrical 
Powar 

4-2.5.  TASK  5.  Inatall  Ruddar  Protractor 

WARNING 

To  praaawt  poaaMo  loath  or  Inliiry  to 
paroaowal  and  damoto  la  agalpntant, 
aS  paraonnat  and  aaaipwiant  nniat  ba 
elaar  at  woaabla  aarfaeaa  whan  tha 
ourfaeaa  ara  Doing  movad. 


2.  liaka  aora  that  aH  paraonnal  and  aquipmant  are 
elaar  e(  movabla  ourfaeaa  par  NAVAIR 
0l-Ft4AAA-2-1.  WP010  00.  Movabla  Surface 
Hazarda. 

3.  EatabHah  aircraft  powar  in  aecordanca  with  Task 
2  (Chaptar  1,  para  1-2.2.). 


4.  At  tha  pilot  atation.  on  tha  loft  aide  eonaoie  c' 
tha  INLET  RAMPS/ENG  CRANK/THROTTlE  cc' 
trol  panal,  oparata  tha  RUDDER  TRIM  awitcn  uni> 
Rig  Pin  ASl8r4S004  can  ba  inaartad  into  the  riQ 
hola  in  tha  dbaetionai  (aai  aaaambiy  (i).  imme 
diataly  balow  earn  (3),  through  tha  rig  pin  holes 
(2)  loeatad  on  both  aidaa  of  tha  directional  fee 
aaaambiy. 


m.  SAMPLE  or  NTIPS  PAPER  TI 


4-11 


®o»T  ovQilable  to  Dl 
P«nalt  iu«r  legible  « 
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IV.  SAMPLE  OP  NTlPaBLBCTRONICALLT-PRBSBNTKD  11 
(REPRODUCED  BY  SCREEN  PRINTER  PROM  SCREEN  DISPLAT) 


NAVAIR  01-F14AAA-2-4^.1  008  03 

15  September  1981  Page  1  of  4 


Preliminary 

ORfiANIZATIONAL  HaNTBIMICC 

RUDDER  PROTRACTOR  INSTALUTION  AND  REMOVAL 

EFFECTIVITY:  Aircraft  Serial  io.  IS7M3  Through  1S7990.  1S8612 
and  Subsequent 


Reference  Material 


fieneral  Aircraft  Information 
Exterior  Safety  Check  .  . 
External  Hydraulic  Power 


NAVAIR  01-F14AAA-2-1 
020  00 
038  03 


Alphabetical  Index 


Subject  Page  No. 

Installation  .  2 

Removal  .  2 


Record  o'^  Applicable  Technical  Directives 
None 


Support  Equipment  Required 


Type  or  Part  No.  Nomenclature 

A51S62060  Rudder  Protractor  (Z 


Materials  Required 
None 


V.  CONVBNTIONAL  WORK-PACKAGE  PAPER  TI 
FOR  A  CORRECTIVE-MAINTENANCE  TASK  (4  PAGES) 


NAVAIR  01*F14AAA-2-4^.1  00803 

Page  2 


1.  IWSTALLATIOW.  (Set  figure  1.) 

M— 

Rudder  protractor  shall  be 
zeroed  (steps  a  through  d) 
before  Installation  on  air* 
craft.  If  tMO  protractors 
are  required,  both  shall 
be  zeroed. 

a.  If  indexer  is  not  secured  to 
protractor,  secure  it  with  four  screws 
provided  on  protractor. 

b.  Loosen  locknuts  that  secure 
adjustment  screws. 

c.  Rotate  adjustment  screws 
clockwise  or  counterclockwise  until 
forward  phenolic  spacers  are  flush  with 
indexer. 

d.  Check  tiiat  indexer  is  at  0. 

If  not.  readjust  screws  until  indexer  . 
is  exactly  at  0.  Secure  adjustment 
screws  with  locknuts. 

e.  Perform  exterior  safety  check 
(NAVAIR  01-F14AAA-2-1,  WP020  00). 

f.  Ensure  that  external  hydraulic 
power  is  disconnected  from  aircraft  and 
that  NO  POWER  placard  is  placed  over 


ground  test  pressure  (OiO  TEST  PRESS) 
fitting  (NAVAIR  01-P14AAA-2-1. 

SUP038  03). 

f.  Raeove  four  screws  (two  on 
do)  from  rear  spar  of  aircraft 

fin. 

h.  Loosen  four  screws,  then 
remove  indexer  from  protractor. 

i.  Secure  protractor  to  fin  wit*- 
four  screws  on  protractor. 

J .  Check  whether  forward  phenol i c 
spacers  are  fully  against  fin.  If  not, 
adjust  knurled  screws  on  protractor 
until  gap  between  spacers  and  fin  on 
both  sides  is  equal. 

k.  Remove  placard  from  6NC  TES*' 
PRESS  fitting. 

2.  REMOVAL.  (See  figure  1 . ) 

a.  Remove  four  screws  that  secure 
protractor  to  fin.  Remove  protractor. 

b.  Install  four  screws  in  res’" 
spar  of  fin. 

c.  Secure  indexer  to  protracto- 
with  four  screws. 
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